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ABSTRACT 


Client satisfaction was considered to be a critical indicator of the effectiveness of 
the services provided by the Navy Regional Data Automation Centers (NARDACSs). 
The purpose of this thesis was to develop the means to measure this indicator. 

Interviews of twenty-eight middle management clients served by NARDACs in 
four geographical regions within CONUS were conducted. Forty-four items that 
influence satisfaction were identified belonging to eight factor dimensions of the 
client satisfaction domain. The Likert Scale methodology was employed in the 
construction of the measurement instrument. The instrument was reviewed for 
completeness of items and logical soundness of operations by three independent 
groups of experts, thus achieving content validity. A client satisfaction index is 
formulated from data obtained with the measurement instrument. 

The instrument was piloted to a NARDAC site consisting of fifty-two client 
organizations using self-administered mail surveys. Evaluation of the instrument 
was performed on the scored results of twenty-six pilot respondents. The pilot 
results were subjected to empirical tests for construct validity and reliability. The 
results indicated that the instrument could be used to measure client satisfaction after 


further piloting on a larger (at least two hundred twenty) sample size. 
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I. INTRODUCTION 


When you can measure what you are speaking about, and express it in 
numbers, you know something about it; but when you cannot express 
it in numbers, your knowledge is of a meager and unsatisfactory kind; 
it may be the beginning of knowledge, but you have scarcely, in your 
thoughts, advanced to the stage of science. [Cited in Ref. 1: p. 89] 


This observation by the renown British physicist, William Thompson (Lord 
Kelvin), may have been foremost in the minds of the Naval Data Automation 
Command (NAVDAC) corporate management during the development of a long-term 
business strategy for their field activities, the Navy Regional Data Automation Center 
(NARDACS). The steering committee, composed of the NARDAC Commanding 
Officers and top management in NAVDAC, was established as the NARDAC Board 
of Directors and chartered to address strategic planning matters and issues affecting 
the NARDACs as a result of a 1986 Coopers and Lybrand study of Navy Industrial 
Fund (NIF) activities.! 

One of the issues that the NARDAC Board of Directors identified in the 
corporate long-range business plan was a need for a standardized index which would 
be used to measure the level of client satisfaction at the various NARDACs. The 
index would serve the purpose of providing an objective evaluation of each 
NARDAC's performance and its ability to deliver quality service to its clients. Also, 


as a measure of effectiveness, the index would be the cornerstone of their business 


1 The Secretary of the Navy (SECNAV) contracted Coopers and Lybrand to conduct 
a management analysis of all activities under the NIF program. The purpose of the study 
was to assess the NIF program's effectiveness and to find any areas of deficiency. 


strategy to increase the NARDAC's competitive posture. This would be 
accomplished by accurately evaluating their current level of client satisfaction to 
enable the establishment of a base index from which management can take 
appropriate actions to make improvements. 

Presently, a standardized measurement of effectiveness that empirically 
represents user Satisfaction among those clients serviced by the NARDACs does not 
exist. Each NARDAC, operating as a cost center, is responsible for marketing its 
services to activities normally located within its geographical area of cognizance. As 
a result, the NARDACs currently assess client satisfaction using different methods, 
criteria, standards and metrics. This, in effect, has made it difficult for NAWDAC to 


obtain an accurate assessment of client satisfaction at each of their NARDACs. 


A. EVOLUTION OF THE REQUIREMENT 
1. Background 

The NARDACs were formed as a result of the Navy Automated Data 
Processing (ADP) Reorganization study and implementation plan with the basic 
objective to improve the management and operation of the Navy's mission support 
ADP program [Ref. 2]. These field activities of NAVDAC were established to 
provide non-tactical ADP technical assistance and operational support to Navy 
activities ashore and afloat, Department of Defense (DOD) components and other 
Federal agencies. There are nine (9) NARDACs geographically dispersed within the 
continental United States (CONUS) and in Hawaii (See Figure 1.1). Although each 
NARDAC may vary in the ADP support it can provide, major functional services are 
offered at every site (See Appendix A). 
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Figure 1.1 NARDAC Installations Map 


2. History of Operations 
Prior to fiscal year 1984, the NARDACS were mission-funded which 
meant that every NARDAC received an annual allotment of appropriated funds to 
cover its costs for the support provided to its clients. Based on that operational 
philosophy, ADP products and services were delivered at no cost to the clients. 
Because this ADP support was perceived by the clients as free, there was little 
concern on their part for costs, only the timeliness and quality of the requested 
support. In that regard, the NARDACS' primary mission was focused on providing 
ADP support to their clients within the planned budgets. The need to formally 
measure client satisfaction, therefore, was not a paramount issue at the time. 
3. Present-Day Operations 
Since fiscal year 1984, the NARDACs' cost of operations has been 
financed under the Navy Industrial Fund (NIF), a revolving fund from which 
working capital funded by Congress to the Department of Defense is allocated to the 
Navy as an appropriation. The initial funds by the NIF appropriation to the 
NARDACs were used to finance the costs of providing ADP services ordered by 
their clients. In return, the products furnished and services rendered by the 
NARDACs to other Navy activities and government agencies are performed on a cost 
reimbursable basis at standardized fiscal year rates.! It is therefore the responsibility 


of the client to submit a Program Objective Memorandum (POM) and budget 


1 Standardized fiscal rates refer to stabilized rates for products and services that are 
fixed for an entire fiscal year. The period of a fiscal year is from 1 October to 30 
September. 


accordingly for needed ADP services.! The clients pay the NARDACs through the 
use of a reimbursable order citing their own appropriated funds. In theory, the 
payment or reimbursement would then put the corpus of the revolving fund back to 
where it started. The financial operations of the NARDACs are comparable to that of 
private enterprises, i.e. working capital and clients are required. The NARDACs and 
other NIF activities, however, are run on a non-profit basis. 

Working with a NIF activity is much like contracting with a contractor 

-- the only significant difference is that the NIF activity is not out to 

make a profit for stockholders. [Ref. 3: p. H22] 
Moreover, the importance of how effective a NARDAC operates cannot be 
understated. The less effective the NARDAC, the higher the charge for services 
[Ref. 3: p. H1). 

According to CDR Charles Taylor, Commanding Officer of NARDAC 

Pearl Harbor, the change to NIF was made for two major reasons. First, in partial 
emulation of enterprises in the commercial sector, the intention was to discipline and 
motivate the NARDACs to provide the best possible service or risk going out of 
business. The second major reason was to provide a measure of discipline to the 
users. If services remained free to the users, as was the case when the NARDACs 
were under mission funding, there would be no incentive for the users to request 
only those services critical for mission accomplishment. Together, these two factors 


were intended to guarantee more efficient use of ADP resources in the Navy. 


1 The POM is a DOD Planning, Programming and Budgeting System (PPBS) 
document, prepared by the Services to the Secretary of Defense, which expresses total 
program requirements in terms of force structure, manpower, material and costs to satisfy 
assigned functions and responsibilities during the period of the Five Year Defense 
Program. [Ref 3: p. A11] 


4. Impact of NIF on Operations 

It was clearly obvious to the NARDACs that the transition to NIF forced a 
significant change in their way of doing business. NIF had introduced new factors 
into the NARDACs' system of operations which had, until then, only applied to 
comparable organizations in the commercial sector. The principal change in direction 
was now focused on the clientele from whom revenues to continue operations were 
obtained. In addition to providing quality ADP support, the issue of operating cost- 
effectively and to remain reasonably affordable for the clients, became of paramount 
importance. Moreover, unlike other NIF activities who maintain a quasi-monopoly 
over the products and services they provide, the NARDACs were now forced to 
compete with other government agencies, as well as the private sector, to provide 
non-tactical ADP support. 

Because of the changes driven by NIF, client satisfaction has become 
critical for the NARDACs. Since all services are provided on a cost reimbursable 
basis and all operating expenses are paid for by revenues generated by the clients, the 
NARDACs must maintain a positive image and provide quality service in order to 
retain their share of the market. A client satisfaction index, therefore, could serve as 
a tool by which the NARDACs can gauge their market leverage and consequently 
make whatever adjustments necessary to remain competitive and financially 


operational. 


B. OBJECTIVES 

The objectives of this thesis are to identify the attributes relating to client 
Satisfaction and to develop a standardized methodology for measuring and analyzing 
these attributes through the construction of a prototype measurement instrument and 


client satisfaction index for subsequent piloting and review at the NARDACs. 


C. FOCAL RESEARCH OUESTIONS 


The focal issues of this thesis are: 


What is an acceptable definition of a satisfied client? 


What attributes should be used to construct a client satisfaction index for a 
NARDAC? 


How should the data be gathered and in what form? 


What metrics should be used to measure and analyze the data and how can 
these metrics be validated? 


Can a microcomputer be used for data analysis and for maintaining 
information on customer satisfaction? If so, what software should be used? 


How can the data gathering and analysis process be institutionalized to make it 
part of an ongoing effort to meet client needs? 


D. SCOPE, LIMITATIONS AND ASSUMPTIONS 


The scope of this thesis is limited to the development of a standardized prototype 


client satisfaction index specifically tailored to represent an empirical measurement of 
client satisfaction for the nine NARDACs. The NARDACs are located in CONUS at 
Naval Air Station (NAS) Alameda - for NARDAC San Francisco, NAS North Island 
- for NARDAC San Diego, New Orleans, NAS Pensacola, NAS Jacksonville, NAS 
Norfolk, Washington Navy Yard - for NARDAC Washington, and Newport; and in 


Hawaii at Naval Station Pearl Harbor. Each NARDAC has a unique set of clients 


within its geographical region. The research focuses primarily on those attributes 


relating to client satisfaction as ascertained from a cross-section of NARDAC clients. 


E. ORGANIZATION OF THE THESIS 


Chapter II presents a literature review pertinent to current thinking regarding 


client satisfaction, in private industry and in government. 


Chapter III describes the methodology employed in the development of a 
measurement instrument designed as the basis for the formulation of the client 
satisfaction index. 

Chapter IV presents an evaluation of the validity and reliability of the client 
Satisfaction questionnaire as a measurement instrument. 

Chapter V describes the construction of the Client Satisfaction Index (CSI) and 
several other methods of analyzing the data collected from the questionnaire. 

Chapter VI presents the conclusions of the study and recommendations for 


implementation and further research . This chapter concludes the thesis. 


II. LITERATURE REVIEW 


A. INTRODUCTION 
The theme of this year's American Marketing Association's annual meeting was 

“The Customer Is King." The meeting focused on, and stressed the importance of 
attracting and retaining customers. Many corporations and management consultants 
have come to the same realization that the key to maximizing profits is maximizing 
customer satisfaction (e.g., [Ref. 4: pp. 2-5], [Ref 5: pp. 13-16], [Ref. 6: p. 5]). 
During the past five years of financial deregulation, banking strategists have 
increasingly concluded that one of a bank's most important assets is its existing 
customer base [Ref. 7: pp. 6, 57]. IBM believes that satisfied customers are the 
basis of their continued business success [Ref. 8]. In the words of Darryl 
Landvater, President of Oliver Wright Video Production, Inc., 

Although there are other ways to get an additional five percent of sales 

to the bottom line, many of them such as cutting expenditures for 

labor, equipment, and research and development are both painful and 

potentially damaging. It's easier to boost sales and earnings through 


Superior customer service and at the same time improve the the long- 
term position of the company. [Ref. 9: p. 86] 


Paul Allaire, President of Xerox, has recently made customer Satisfaction it's 
“first priority”, with return on assets (ROA) and increased market share second and 
third respectfully [Ref. 4: p. 2]. In a memo to his division directors, Allaire says, 
“we can only achieve our ROA and market share goals by satisfying our customers." 
Xerox's new focus on customer satisfaction was the basis for the cover story, 
“Customer Satisfaction: The Big Payoff" in Xerox's quarterly corporate publication 


Benchmark [Ref. 4]. In this article, five leading management consultants expound 


on the importance of satisfying the customer and its payoff in increased profits. In 
the article, Robert Waterman, Jr., co-author of In Search of Excellence, cited a study 


from the Strategic Planning Institute in Cambridge, Massachusetts. 


Business sectors with a higher service content have a higher return on 
investment. The institute, which ranked similar companies in order by 
the relative quality of their products and services, found that the 
highest rated company in each area had a return on investment 100 
percent higher than the lowest-ranked company. [Ref. 4: p. 3] 


Christopher Lovelock, a former Harvard professor and principal of Christopher 
Lovelock and Associates, states that the key to improving customer satisfaction "...is 
with research to measure what characteristics of services or products are important to 
users as well as how customers perceive the goals provided by the company.” [Ref. 
4: p. 3] 

Tom Peters, co-author of In Search of Excellence and A Passion for Excellence, 
believes that the qualitative aspects of business must be quantified. He believes in the 
old adage "What gets measured gets done.” This philosophy is applied rigorously 


within his organization and at his executive seminars in the following manner. 


In the customer arena, we believe that regular quantitative measurement 
of customer satisfaction provides a much better lead indicator of future 
organizational health than does profitability or market-share change. 
We suggest monthly measurement. Further, we urge participants to 
make the level of customer satisfaction the primary basis for incentive 
compensation and annual performance evaluation for virtually every 
person at every level in every function throughout the organization. 
[Ref. 10: p. El] 


Clearly, customer satisfaction is an issue which is receiving considerable 
attention in the corporate community and recently it has been gaining momentum in 
the computer services industry. The purpose of this literature review 1s to survey 


some of the recent research that has been conducted in the area of customer 
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satisfaction with computer services and the instruments which were developed to 
quantify and measure satisfaction. The literature review is divided into three 
sections. The first section, Measures of Computer User Satisfaction, is a review of 
generic measures which could be used at any computer services organization or 
management information system/data processing (MIS/DP) department. The second 
section, Survey of Current Industry Measures, is a review of current measures of 
customer satisfaction used in the computer industry. And the last section, 
Department of Defense Measures, is a brief look at what kinds of measures of 


customer satisfaction are being used within the Department of Defense. 


B. MEASURES OF COMPUTER USER SATISFACTION 

The largest initial obstacle in conducting this literature review was finding 
information related to the measurement of customer satisfaction at a computer 
services organization. The problem was the word "customer". The computer 
industry and MIS/DP departments, in many instances, still refers to its customers as 
users instead of valuable customers [Ref. 11: p. 286]. According to W.H. Inmon, 
“Universally, data processing exists as a service organization within the company. 
The service is for the user, and the satisfaction of the user ultimately determines the 
success or failure of data processing.” [Ref. 12: p. 224] 

There have been several studies conducted recently on the concept of computer 
user satisfaction or user information satisfaction (UIS). In the words of livari, UIS 
refers to "...a cluster of concepts that imply the assessment of information systems or 
information services in the user's subjective terms." [Ref. 13: p. 57] The 
foundations of UIS are based on the research by Cyert and March [Ref. 14]. The 
essential concept is "...that an information system which meets the needs of its user 


will reinforce satisfaction with that system. If the system does not provide the 
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needed information, the user will become dissatisfied and look elsewhere ." [Ref. 15: 
p.786] 

One of the first measures of UIS related to the total MIS/DP function in an 
organization which has received considerable attention was developed by S. W. 
Pearson [Ref. 16]. During his research, Pearson compiled a list of thirty-six factors 
relating to computer user satisfaction based on a literature review of twenty-two 
studies of computer-user interactions. These factors were then reviewed by three 
DP professionals for completeness and accuracy. As a result of the review, two 
additional factors were added to the list. Next, the list of thirty-eight factors was 
compared, using a critical incident analysis technique, to taped interview responses 
from thirty-two middle manager users in eight different organizations. This step 
concluded with the addition of one factor, making a total of thirty-nine distinct 
factors. These factors were then incorporated into a questionnaire which utilized the 
semantic differential technique. Each factor is measured by three separate scales 
using seven intervals from negative to positive responses. The first scale was a 
measure of one's perception by rating four bipolar adjective pairs. The second scale 
was a Satisfactory - unsatisfactory pair and the third scale measured the importance of 
the factor to the user. The importance rating is used as a weighting factor for the 
overall satisfaction score. The description of the scoring method can be found in 
Reference 17. The questionnaire was empirically tested for validity and reliability 
using the same thirty-two middle managers previously interviewed. The instrument 
was found to be reliable and valid, based on the twenty-nine returned questionnaires. 
See Reference 17 for the results of the validity and reliability evaluation of the 


questionnaire. 
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A study by Ives, Olson and Baroudi was a continuation of the Pearson study 
with the emphasis to reduce the length of the overall measure and reinforce the 
validity of the instrument with more extensive testing (Ref. 15]. The results of their 
study reduced Pearson's thirty-nine original factors down to twenty-one factors with 
greater validity using a larger sample size. Description of their results can be found 
in [Ref. 15]. Ives, Olson, and Baroudi's work was based on the following 
definition of UIS: ”...the extent to which users believe the information system 
available to them meets their information requirements." 

Mathew & Co., a Data Processing Management consulting firm, has developed a 
reporting system for measuring DP-user satisfaction called "How're We Doing?" 
(Ref. 18: p. 10] The system requires users to annotate in a log, during a one month 
period, any problems with data processing based on nine "performance-related 
criteria." At the end of the month, the data from the log(s) are processed by Mathew 
& Co. which supplies the client with DP user satisfaction analyses in several 
different categories. The system is based on exception reporting (i.e., only problems 
with data processing are reported). The nine criteria are timeliness of output, quality 
of output, online availability, response time, systems development schedules, 
response to problems, and attitude and cooperativeness. Mathew & Co. provides its 
clients with reports on DP-user trends for the current month and trends over periods 


of three months, six months and twelve months. 


C. SURVEY OF CURRENT INDUSTRY MEASURES 
The banking industry recently began using a customer service index to measure 
and manage customer service [Ref. 7]. One index is based on twenty critical 


attributes of "Good Customer Service” according to a nationwide survey of bank 
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customers. Bank customers were asked to score all twenty attributes of their 
financial institution on a six- point scale, (6 = excellent).! 

Keeping the customer satisfied with quality products and services is the 
challenge changing the face of the automotive industry also. The automotive industry 
uses a consumer satisfaction index to evaluate consumer demands (e.g., increasing 
showroom hours). [Ref. 19] 

Sitmar Cruises attributes its highest rate of repeat passengers of any major cruise 
line to its "intensive customer research” using a 16-page customer survey [Ref. 4: p. 
4]. Benjamin French believes random surveys are the best way of determining the 


effectiveness of an organization's customer satisfaction program. 


A brief questionnaire and accompanying letter of explanation are 
simple to prepare and provide an inexpensive way of taking the pulse 
of your customers. In addition to providing you with data on your 
effectiveness, the survey also conveys your concern for solving 
customer's problems and gives you another opportunity to restore their 
faith in your company and its products or services. [Ref. 20: p. 111] 


However, James Carman, Professor of Business Administration at the Graduate 
School of Business, University of California at Berkeley, argues that surveys must 
be interpreted carefully because it is difficult for some customers to evaluate the 
quality of the service they received if they are lacking knowledge in the field. [Ref. 
4: p. 5] 

IBM distributes a survey to every one of their customers at least once a year in 
order to directly ascertain information relating to customer satisfaction and to improve 
their business relationship. The corporate offices send out periodic surveys to clients 


requesting their service perceptions. This is done independently from the regional 


I The index was developed by Financial Products Group, a Chicago-based consulting 
firm serving the financial services industry, with the assistance of Market Facts, inc. 
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offices. In fact, regional offices don't even know when this occurs. The survey is 
divided into eight major categories: guality of products, hardware maintenance and 
support, systems management, marketing support, technical support, education, 
telephone coverage, and overall rating. The survey is scored on a five-point Likert 
Scale (very satisfied to very dissatisfied) with a block for no opinion. In addition to 
the survey, there is a corporate-wide complaint system where client complaints are 
assigned to a high-level manager for investigation and follow-up. [Ref. 8] 

At Electronic Data Systems Corporation (EDS), they believe feedback from the 
customer is an important step in maintaining a high level of customer satisfaction. 
EDS North American Commercial Group uses an extensive series of questionnaires 
as part of their Quality Enhancement Program (QEP)! to keep abreast of their 


customers needs and to continue to improve services and products. [Ref. 21] 


D. DEPARTMENT OF DEFENSE MEASURES 

The Philadelphia Naval Shipyard, a NIF-funded activity, does not use any 
proactive initiatives to ascertain customer satisfaction. Their criteria for measuring 
performance is based on adhering to project budget and schedule constraints*. The 
Military Airlift Command (MAC), an Air Force industrial funded activity, does not 
actively solicit information from their customers regarding customer satisfaction 


either. Instead, suggestion/comment forms are made available to MAC customers.3 


1 The contents of the questionnaire and the specifics of the QEP implementation 
policies and procedures are proprietary information of EDS North American Commercial 
Group and will not be disclosed in this thesis. 


2 Interview with Mr. Monteleon, Management Planning Division, Philadelphia Naval 
Shipyard, Philadelphia, Pennsylvania, 5 February 1988. 


3 Interview with CDR Jordan, USN, Navy Liaison Officer, Military Airlift Command 
Headquarters, Scott Air Force Base, 5 February 1988. 


15 


The Navy Public Works Centers (PWC), which are NIF-funded activities, have 
recently initiated a customer satisfaction evaluation program [Ref. 22]. The program, 
“Customer Evaluation of Support Provided by PWC", consists of a fifty-item 
questionnaire divided into six categories. The respondent is requested to rate the 
quality of service provided by the PWC using a seven-point scale. The questionnaire 
is scored in the following manner: 


a. For customer responses where the "X" is placed in blocks 6 or 7, 
score 2 points; 


b. For customer responses where the "X" is placed in blocks 3, 4, or 
5, score 1 point; 


Cc. For customer responses where the "X" is placed in blocks 1 or 2, 
score 0 points. 


The best possible score is 100 points. To determine the annual grade 

from all of the customers who received a guestionnaire, add the total 

guestionnaire scores and divide by the total number of guestionnaires 
that were completed in the year. 'The total score is weighted by 
customer size (i.e. four surveys from large customers, two from 

medium- sized customers and one each from all others.) The result is a 
summarized customer evaluation of the support provided by the PWC 

in the past year. 


We found no published studies performed by the Department of the Navy or the 
Department of Defense in the area of customer or user satisfaction relating to 


computer services organizations or MIS/DP departments.! 


I Searches were made through the Defense Logistics Studies Information Exchange 
(DLSIE), Defense Technical Information Center (DTIC), and DIALOG Information 
Services, Inc. 
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HI. OUESTIONNAIRE DEVELOPMENT 


The corporate heads of the NARDACS identified the critical reguirement for a 
client satisfaction index, thus setting the stage for the next step of our research. We 
needed to identify a logical process to follow in order to arrive at an index that 
accurately represents the client's satisfaction with the NARDAC's performance in 
providing products and services. 'To represent that state of satisfaction as 
unequivocally as possible, it was imperative that the theme of our methodology be 
based on the inputs from actual NARDAC clients and their composite view of client 
satisfaction. This was important because it ensured that the fundamental building 
blocks for the construction of a meaningful index were obtained directly from 
representative samples of the source. By using this approach, we felt that our 
methodology would be sound in terms of face validity and the resulting product 
would be realistic and more readily accepted by the NARDACs and their client 
community. 

From a study by Pearson, the measurement of client satisfaction mirrors the 
measurement of an individual's attitude or evaluative feelings toward some feature of 
the subjects of interest [Ref. 16]. There are various ways to collect data for 
measuring a client's attitude. One basic means is by interrogation which involves 
getting people to answer questions, either in person or through telephone interviews. 
Another interrogation technique, more widely used for measuring a person's attitude, 
involves an instrument, in most cases a questionnaire, which is one of the 
fundamental tools used in survey research. The reason for the popularity of the 


survey research method is due to the numerous benefits it offers. 
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e It provides a method for empirical verification of data. 


« The data gathered by this method becomes a source of information which can 
be conveniently stored through automated methods and can be analyzed shortly 
after collection. 


+ The responses can be coded into a standardized form for recording in a 
quantitative manner; therefore a standardized measurement that is consistent 
across respondents is achieved and lends itself to the development of an index. 


e Surveys can be administered fairly readily by an implementing organization (a 
NARDAC) and can be flexibly designed with minimum impact on the 
respondents’ (the clients’) time. ([Ref. 23] and [Ref. 24]) 

It is because of these advantages and its applicability to the measurement of client 


satisfaction that the survey research approach was selected for this study. 


A. BACKGROUND å 

During the preliminary stages of the research, background information on the 
NARDACS was gathered in order to obtain an understanding of their organizational 
structure and hierarchy, corporate culture and environment, products and services, 
and client information. This background information was obtained during site visits 
at NARDAC San Francisco, NARDAC Washington and NAVDAC. Further data 
was received from NARDAC Pearl Harbor and NARDAC San Diego. Additionally, 
the marketing representative at Honeywell was interviewed to see how client 
satisfaction was viewed from the industry perspective. We also reviewed literature 
on customer satisfaction and found current trends in the field, particularly in service- 
oriented industries. 

At the same time, we were given information on existing customer satisfaction 
methods employed by the NARDACs and Public Works Centers (PWC). Primarily, 
the methods used by these communities are informal and based on feedback from the 
client, usually obtained during personal visits or phone conversation. The PWCs 


and several NARDACs employed more formal methods, using questionnaires that 
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were developed in-house, as part of their marketing efforts. Some of these methods, 


extracted from a summary of findings [Ref. 25], are listed below. 


e NARDAC Jacksonville developed a client assessment form which is used 
during quarterly client visits. This assessment seems to be the most formal 
means employed to gather client satisfaction information from the client's 
perspective. 

e NARDAC San Diego identified specific client satisfaction indices with it's 
mainframe clients. These indices are: "95% products on time” and "98% on- 
line availability during prime time and off-shift hours.” Prime time availability 
is particularly significant to its major client, the Naval Aviation Depot. These 
indices are assessed and reported during regular client meetings. 


« NARDAC San Francisco recently developed an information form which will 
be used by client relations personnel when visiting clients. A portion of this 
form provides clients the opportunity to express satisfaction with or concerns 
about services. 


e NARDAC Pensacola and NARDAC San Diego use written reports of visits or 
phone contacts to identify client problems or concerns. These reports also 
provide vehicles to convey the client's satisfaction with services. 


¢ NARDAC Washington developed a quarterly rating report which is filled out 
by NARDAC Washington department directors. This report provides the 
department directors’ expectation of a client's response to various service 
Satisfaction questions. 


Because of the wide diversity in methods used among the NARDACs, it is 
understandable why the NARDAC Board of Directors recognized the need for a 


standardized method of obtaining critical client satisfaction information. 


B. FACTOR IDENTIFICATION 

| Using the information gathered from interviews with NARDAC management 
personnel, relevant organizational documents and literature searches, we generated a 
list of the major products and services that the NARDAC provides to its customers. 


From this list, an initial set of factors, which was seen as measuring the NARDACS 
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performance in delivering products and services, was compiled. This initial set 


consisted of the following nine (9) factors: 


e Computer capabilities 

e Ease of obtaining services 

e System design and programming expertise 

e Technical support 

+ ADP consultation and assistance 

e Training 

e Cost of ADP services 

e Customer support 

e Timeliness and quality of products and services provided 


In contrast, the factors used in existing NARDAC questionnaires [Ref. 25] included: 


e Response to client requests (the quality of response and the rapidity of 
response) 


e Thoroughness of information conveyed to clients 

e NARDAC service accessibility to clients 

e Cost 

e Perception of the reliability of NARDAC personnel 


e Non-recurrence of problems and willingness to refer other organizations to 
NARDAC for computing services 


C. CONTENT VALIDATION 

In order to validate the accuracy of our initial set of factors and thereby establish 
content validity, a method was needed which was expedient, allowed for personal 
interaction with existing clients to verify the assumptions made in compiling the list 
of factors, and permitted the opportunity to identify other aspects of the NARDAC- 
client relationship which may have been overlooked or omitted during previous 
interviews with NARDAC personnel and documentation review. It was decided then 


that interviews with a representative group of active NARDAC clients would 


accomplish this objective. A series of interviews with clients served by the 
NARDACS would be conducted within research limitations. 
1. Interview Procedures 

Because of time constraints and a ceiling on travel expenses, it was 
necessary to set criteria for determining our interview sample. This criteria included 
location, types of client organizations to interview, who within the client organization 
to interview and number of interviews to conduct. 

The first and foremost criterion was location. We wanted to focus on 
regions having a large concentration of Navy activities with differing ADP support 
requirements and where major NARDAC installations supporting these Navy 
commands offered a wide variety of computer services. The East Coast and West 
Coast regions appeared to satisfy this criterion and offered the best representative 
cross-section of respondents. The clients to be selected for interview were those 
supported by NARDAC San Francisco and NARDAC San Diego on the West Coast 
and NARDAC Washington and NARDAC Norfolk on the East Coast. 

The next criterion was types of clients to consider for the interviews. Since 
the objective in this phase of the research was to obtain information on what factors 
affect or influence client satisfaction, we wanted to interview current/active clients 
who maintained some frequency of interaction with the NARDAC providing them 
services. One way of determining the relative degree of interaction was to examine 
the amount of revenue generated from the client during the last fiscal year. This data 
was readily available to the Client Liaison Officer at each NARDAC who compiled a 
ranked listing of clients (by revenues earned) from which candidate clients were 


selected. Although interviews with previous clients who had elected not to renew 
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services with the NARDAC were considered, we were unable to include that client 
category under this criterion because of time constraints. 

Another criterion used in narrowing the sample size of clients for the 
interview was selection of the person in the client organization with whom to conduct 
the interview. We determined that the best candidate to satisfy this criterion was the 
individual designated by the client organization as the command's official liaison or 
point of contact (POC) with the NARDAC. This individual would usually belong in 
middle management, serving as the activity's ADP Officer. Since POC information 
normally resided alongside the client organization data, this information was 
maintained and kept current by the NARDAC's Client Liaison Officer. 

The last criterion, number of interviews to conduct, was constrained by a 


number of factors: 


e Number of days available to conduct the interviews 
e Availability of the clients for the interview 
+ Length of the interview 


e Travel time window required between interviews (during normal working 
hours) 


In most cases, the Client Liaison Officer of the NARDAC was in a better position to 
juggle these constraints and therefore coordinated appointments with the clients and 
arranged the interview schedule. This schedule included interviews with 28 client 
organizations, the list of which is summarized in Appendix B. Each of us 
interviewed the client organization's POC, some of whom brought in members of 
their staff to provide input during the session. 

After the appointments with the clients were scheduled, the Client Liaison 


Officer provided us a list containing the names and addresses of the clients to be 


interviewed. Shortly thereafter, we contacted each client by phone to personally 
introduce ourselves and to explain the nature of our research. Following these phone 
calls, we mailed each client a letter containing the purpose of the interview, in 
particular, to obtain their views regarding the factors that they, as clients, would 
use to evaluate the services provided by their NARDAC. A sample letter can be 
found in Appendix C. A week before the interviews, we again called each client to 
verify receipt of the letter and to confirm the date and time of our interview. This call 
also gave the client the opportunity to ask specific questions about the interview or 
our research and allowed for any last minute rescheduling of the appointment if 
required. 

To ensure efficient use of time during the interview and to facilitate data 
collection/recording, forms were used which we had prepared beforehand to guide us 
through the interview. These forms served to assist us in collecting background 
information regarding the clients (see Appendix D) and their views on what particular 
factors contributed to their satisfaction with NARDAC services (see Appendix E). In 
addition, the forms were used to ensure consistency in the types of information asked 
of the client. In certain instances, interviews were taped but not before receiving 
permission from the client. In no instance was permission denied. 

2. Factor Validation 

During the interviews, we asked each client to comment on the nine factors 
which we had previously compiled during the process described in Section B and to 
indicate whether each factor influenced their satisfaction with the NARDAC's 
performance in providing services. The client was also asked to identify any other 
factors which were not included in the original set and did influence their state of 


satisfaction. We then requested each client to rank all the factors by order of 


importance. In some cases, clients were strongly inclined to group several factors 
equally important under the same ranking. Other clients felt that some of the factors 
were not pertinent to their current requirements and therefore excluded those factors 
from their ranking. 

Following the completion of all the interviews, we compiled a tabulated list 
of factors that were ranked important to client satisfaction by each client. The 
compiled list provided us an across the board ranking of all the factors to help in 
determining which factors were considered most important by the clients. To obtain 
a composite ranking of the factors, we averaged the rankings given by the clients on 


each factor. The composite ranking of the top six factors included: 


Ranking Factor 
l Timeliness and quality of products and services 
2 Technical support 
3 System design and programming expertise 
4 Ease of obtaining services 
5 Customer support 
6 Cost of ADP services 


We ascertained that the highest ranked factor was, in fact, two separate 
factor dimensions, Timeliness and Quality. By incorporating these dimensions as 
separate factors, a modified list of seven factors was obtained. The remaining factors 
in the original set were not discounted. They were determined to be too specific in 
scope and were therefore considered to be candidate subelements (items) relating to 


one of the seven factors. 


3. Item Development and Validation 

Our next step was to identify specific items obtained from the interviews 
which related to the NARDAC's performance and which were measurable. The 
information was extracted from client comments documented on the form we used 
during the interviews (see Part 4 of Appendix E) and obtained in taped sessions. 
Working with our own individual set of interview data, we placed each item ona 3 
by 5 card. When this was completed, we compared our sets of items for redundancy 
and clarification, while referring to information from other surveys and relevant 
literature. Our sets were then consolidated into a set containing forty-two 
performance-related items which were then categorized under one of the seven 
factors we had identified in the previous process. In those instances when items 
could not be placed under a category because of the factor’s specificity, we redefined 
the scope of the factor and broadened its applicability to include the item. The 


factors, as modified by this process, are listed below. 


e Timeliness of Products and Services 

e Quality of Products and Services 

e Technical Proficiency 

+ Hardware, Software and Communications Technology 
e Accessibility 

e Customer Support 

e Service Level Management 


The final set of items and modified factors established the client satisfaction 
domain for the NARDACs and provided the fundamental basis of what to measure in 
order to gauge the level of an individual client's satisfaction. The key elements of 
this set, which is presented in Appendix F and discussed in the following sections, 


formed the foundation for the construction of the measurement instrument. 


D. MEASUREMENT INSTRUMENT 

Several different measuring instruments were investigated through literature 
searches and reviewing surveys currently being used in the government and private 
industry to determine which instrument would best suit our study. Buzzell, Cox and 
Brown assert, the more structured the measuring instrument is, the more accurate the 
output [Ref. 26]. Typically, in marketing measurements, it is important to have a 
measure of the degree of the response. To aid in this type of measurement, degrees 
of response are scaled to make more or less discriminations. Three of the most 
common marketing measurements are variants of the Semantic differential scale, 
Likert scales and Paired comparisons (see Figure 3.1). According to Buzzell, Cox 
and Brown, however, 


Although it is possible to attach numbers to such a verbal scale for 
purposes of analysis, it should be remembered that this is still an 
ordinal scale, strictly speaking, and estimates with regard to intervals 
or degrees of response are simply estimates. Quantify them if you 
will, but there is no direct means of determining the validity of such 
quantifications. 


There are several methods of collecting data for the instrument. The most 
common ones used in marketing research are personal interviews, telephone 
Interviews and self-administered questionnaires (mail surveys). Listed in Figure 3.2 
below are some of the advantages and disadvantages of each method, as cited by 
([Ref. 27] and [Ref. 28]). The surveys that we found currently being used in the 
government and industry were all self-administered questionnaires. In the 
government, cost and time restraints seem to make self-administered mail 
questionnaires more prevalent. 

1. Questionnaire Development Standards 

Due to the limited resources available ata NARDAC to conduct surveys, 


self-administered questionnaires would be the easiest to implement. There are 


SEMANTIC DIFFERENTIAL SCALE 


"Would you rate General Motors as being: 


Progressive Conservative 


LIKERT SCALE 
“Ajax is an excellent cleanser." 


Agree extremely strongly 
Agree fairly strongly 


Agree 

Undecided 

Disagree 

Disagree fairly strongly 
Disagree extremely strongly 


PAIRED COMPARISONS, e.g., 


Do you prefer Brand A or Brand B? 
Do you prefer Brand C or Brand A? 
Do you prefer Brand B or Brand C? 





Figure 3.1 Commonly Used Types of Rating Methods and Scales. 


1. Personal interviews : 


Advantages: 

e They tend to allow the use of a more representative sample. 
Achieve higher return rate. 
Produce fewer incomplete questionnaires. 
More questions can be asked. 
More complex measurement methods can be used. 
Verification or responses may be made more readily. 


Disadvantages: 
e More costly. 
e Subject to interviewer bias, error and cheating. 
e Subject to response bias. 


2. Telephone interviews : 


Advantages: 
e Can be conducted quickly. 
e Relatively low cost. 


Disadvantages: 
e Sample bias. 
e Usually must be brief. 


3. Self-Administered questionnaires: 


Advantages: 
¢ Least costly. 
e Avoid interviewer bias. 
e Larger number of respondents can be reached. 
e More convenient to the respondent. 
e Requires a smaller staff for administering. 


Disadvantages: 
e Sample is almost certainly not likely to be “representative unless follow 
up is done. 
e Must be carefully designed and pretested to avoid confusion on the part 
of the respondent. 


Figure 3.2 Data Collection Methods (Advantages and Disadvantages) 


several methods of conducting a self-administered guestionnaire. The guestionnaire 
could be administered to a group of respondents gathered at the same place at the 
same time (e.g., at a NARDAC Client Council/Advisory Board meeting). They 
could also be hand-delivered at the respondent's office to be completed and picked up 
at a later time. Mailing is another option, or a combination of mailing and one of the 
above methods can also be used. [Ref. 23: p. 159] The questionnaire can be hand- 
delivered, where it could be explained and any questions answered, and then mailed 
when completed. The questionnaire designed for the NARDACs was based on the 
premise that the survey would be mailed with limited intervention required by the 
NARDAC. The following criteria was used in developing the NARDAC Client 
Satisfaction questionnaire: 

e Instructions for completing the questionnaire must be clear and concise. 

e Scales must be easily understandable and unambiguous. 

¢ Questions will be closed-ended and as easy to answer as possible. 

e Definitions will be provided as required. 

e Comment section will be provided. 


+ The number of questions should be held to an absolute minimum. | 


e Questionnaire must have an overall professional appearance (i.e., neat and 
legible). 


Since the majority of the questionnaire respondents are considered middle 
to upper level management within their organization (GS-12 to GM-15), we felt a 
self-administered questionnaire would provide us with the needed information with 
minimum intrusion on their schedules. In this regard, we felt it was important that 
the instructions were self-explanatory and the questions easily answered. According 
to Arlene Fink, "A self-administered questionnaire that is hard to read can confuse or 


irritate respondents. The result is a loss of data.” [Ref. 28: p. 44] We tried to 


formulate the guestions in the respondents' own vernacular. Definitions of certain 
terminology and phrases were provided to help alleviate any ambiguities. We also 
felt it was important to provide the respondent with an opportunity to comment on 
any of the items in the questionnaire or provide additional information since the 
questions were closed-ended. Additionally, the questionnaire had to make a good 
visual impression. Since the questionnaires are distributed to valued NARDAC 
clients, we felt that if the appearance made a bad first impression, then response rate 
could possibly suffer. 
2. Questionnaire Format 

The questionnaire measures the respondents level of satisfaction with respect to a set 
of forty-two performance related items. These items are grouped together into seven 
categories or factors as described in section B (Factor Identification) above. 
Although we were concerned with the length of the questionnaire, we allowed 
adequate space between questions for comments and to prevent the questionnaire 
from looking cluttered. The questionnaire is eight pages long or four pages copied 
back-to-back. It is widely acknowledged in the social research community that an 
improperly laid out questionnaire can not only confuse the respondents, but also 
make the scoring of the questionnaire more error-prone and time consuming. One of 
the most common questionnaire formats is one where the respondent is asked to 
check one response from a series [Ref. 27: p. 205]. Each item is scored using a five- 
point Likert scale. A not applicable box is also provided. According to Babbie, 
boxes, adequately spaced apart, are the best for the respondent to answer. The scale 
is measured from one to five, where five is very satisfied, and one is very 
dissatisfied. The respondents are asked to place an X in the box which most 


appropriately describes their level of satisfaction with that item. The scales are laid 


out vertically to the side of each item to facilitate scoring of the guestionnaire. Key 
words and/or phrases were underlined in each item to help focus the respondent's 
attention on the major point the item is measuring (see Appendix F). 
3. Questionnaire Critique 

After the first draft of the questionnaire was completed, it was sent out for 
review to all nine NARDAC Client Liaison Officers/Marketing representatives, and to 
various Naval Postgraduate School faculty (see Appendix C). Generally, the 
comments from the NARDACs were favorable. Most felt we had developed a 
comprehensive questionnaire and had addressed the major issues presently 
concerning the NARDAC clients. The majority of the comments focused on the 
format and length of the questionnaire. Although some of the NARDACs indicated 
there were some questions that might be redundant, none of them indicated which 
ones were the culprits. One NARDAC felt the sentence structure of the items was 
too choppy and short, and another suggested we include an overall rating question at 
the end of the questionnaire. Surprisingly, the majority of the comments regarding 
item construction came from the Naval Postgraduate School faculty. It was pointed 
out that many of our items were asking the respondent to rate more than one aspect in 
a single item. For example, in item twelve, the respondent is asked to rate the format 
and the quality of the Chargeback Report. The respondent may feel that the format is 
satisfactory, but the quality is not. Other comments dealt with the lack of definitions 
regarding the meaning of certain terms, and the order in which the questions were 
presented. The above comments were incorporated into the second draft of the 
questionnaire. Comments referring to the implementation of the questionnaire were 
disregarded because they fell outside the scope of our study (e.g., adding an internal 


routing stamp). 
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4. Questionnaire Second Draft (Short Form) 

The second draft of the questionnaire looked considerably different from 
the first version (see Appendix G). It is divided into three major sections. The first 
section still measures the respondents level of satisfaction with respect to a set of 
performance related items, but the three comment lines have been dropped, two 
additional questions were added, and the ordering and grouping of the questions 
were changed. These revisions made the questionnaire appear to be shorter - four 
pages long or two pages copied back-to-back (Short Form ). The items are still 
grouped together into seven categories or factors, but the names have changed. 
Although the length of the questionnaire was a concern, we did allow adequate 
spacing between questions to prevent the questionnaire from looking cluttered. The 
response boxes formed a matrix down the right side of the page to facilitate the 
scoring of the questionnaires. The scale was repeated at the top of each page for the 
convenience of the respondent. 

A new section was also added to the Short Form. In Robert Alloway's 
paper, Defining Success for Data Processing, he discovered that in the six companies 
that he studied, all were violating the fundamental rule of management, "...identify 
which activities are most important and allocate resources to ensure good 
performance on those activities.” [Ref. 29: p. 1] Thus, the second section of the 
guestionnaire measures the relative importance of each of the seven major categories 
described in the first section of the questionnaire. Again, each category is scored 
using a five-point interval scale. Boxes are provided for the respondents answers. 
The scale is measured from one to five, where five is critical, and one is irrelevant. 
The use of the second section will be described in detail in Chapter V. The data from 


this section will not be used in the computation of the index; it will only be used as a 
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management tool. The last section of the questionnaire permits the respondent to 
make comments about any item(s) or general comments about the service provided 
by the NARDAC. The trailing edge of the questionnaire provides the NARDAC with 
pertinent administrative information regarding the respondent. 
5. Final Review of Questionnaire 

Å revised long form ) and the short form were sent out to all NARDAC 
Commanding Officers (COs) and NAVDAC for one last review before the pilot 
survey was conducted. During a NARDAC Board of Directors meeting, the COs 
decided that the scale identifiers at each extreme did not demonstrate enough 
differentiation from the other identifiers. Therefore, the words extremely satisfied 
and extremely dissatisfied replaced the identifiers very satisfied and very 
dissatisfied. They also changed the middle identifier from Neither Satisfied Nor 
Dissatisfied to Neutral. They felt this wording would be be easier to interpret by the 
respondents and NARDAC management. Other changes that were made included the 
definition of factors and items, the rearranging and regrouping of items, and the 
revision of some of the factor names. One additional factor was also added. The 
short form of the questionnaire was unanimously approved. They felt that the long 
form would be too overwhelming for most of their clients. However, they still 
would like to use the long form for specific clients from whom more data is required 


to help identify and resolve discrepancies. 


l The revised long form is the original questionnaire with only content changes 
made. The format remained the same (see Appendix H). 
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IV. OUESTIONNAIRE EVALUATION 


This chapter describes the methods we employed to evaluate the measurement 
instrument, the client satisfaction questionnaire. As discussed in the preceding 
chapters, this instrument was developed to capture the domain of client satisfaction 
for the NARDACs and the distinguishable elements (items) that influence this 
domain. The administration of the questionnaire to a group of subject clients at a 
pilot site and the evaluation of the questionnaire based on the scored results are 
presented. The results were analyzed using several statistical software packages. 
STATGRAPHICS, a personal computer (PC) based statistical graphics system, and 
the Statistical Package for Social Scientists (SPSS-X), a mainframe computer 
program, were used in performing descriptive statistics, construct validation tests, 
and reliability tests. In addition, various microcomputer programs were augmented 


to provide other tools for the evaluation of the measurement instrument. 


A. PILOT DESCRIPTION 

To collect the data for evaluation of the questionnaire, we used a self- 
administered mail survey. One NARDAC site was chosen to pilot the measurement 
instrument, consisting of fifty-two client organizations which formed the sample 
population. Although we realize that a nationwide survey would provide us with a 
greater sample size to item ratio and more statistically sound test results, time and 
schedule constraints and the lack of physical resources compelled us to preclude 
exercising this alternative. Nevertheless, we felt the pilot sample size was sufficient 


for an initial evaluation of the instrument. Considering geographical proximity, the 


clients serviced by NARDAC San Francisco were therefore selected as the subject 
group for evaluation of the guestionnaire (see Appendix D. 
1. Pilot Administration 

After receiving approval from the Commanding Officer of NARDAC San 
Francisco to conduct the pilot, we obtained a list of active clients from his Client 
Liaison staff. Information from this list included the name of the client organization, 
name of the organization's POC, mailing address and telephone number. 
Additionally, the staff supplied us with supplemental profile information on each 
client. The profile data consisted of current fiscal year revenues earned to date from 
the clients and types of services provided to the clients by the NARDAC. 

Once the NARDAC Board of Directors gave us the approval to proceed 
with the pilot testing, we mailed each client a pilot survey package that consisted of a 
cover letter, the self-administered questionnaire (see Appendix J) and a pre- 
addressed return envelope. In the cover letter, we requested each client's cooperation 
in completing the questionnaire to assist us in evaluating the measurement 
instrument. The client was also asked to critique the questionnaire in terms of 
content, presentation and format. A sample of the cover letter is presented in 
Appendix C. 

The survey packages were posted to the clients ten days before the 
requested return deadline which allowed the respondents at least one full work week 
to complete and return the questionnaire. The return envelope was provided to 
accelerate the return time. 

2. Pilot Response 
Of the fifty-two survey packages mailed, we received twenty-six completed 


questionnaires which seemed to suggest a fifty percent response rate. Current survey 
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research guidelines on percentage return rates indicate that fifty percent is an 
acceptable response rate. Babbie suggests that a response rate of seventy percent or 
more is very good, a response rate of at least sixty percent is good and a response 
rate of at least fifty percent is adequate for analysis and reporting. He does caution, 

however, that this yardstick is only a rough guide which has no statistical basis. The 
accepted practice in computing response rates is to omit all those questionnaires that 
could not be delivered due to bad addresses and the like. This number is subtracted 
from the initial sample size to obtain the net sample size. Then, the number of 
completed questionnaires is divided by the net sample size to produce the response 
rate. [Ref. 23: p. 165] Although the number of questionnaires we received was far 
below the number we expected, we felt that this could have been attributed to the 
timing of the pilot survey which occurred during the Christmas season (early 

December). To substantiate reasons for adjusting the net sample size, it was 
necessary for us to obtain concrete evidence for the number of omitted non- 
responses. To accomplish this, we made numerous attempts to contact by phone 
those clients who had not responded. In many cases, we were unsuccessful in our 
efforts due to busy signals or no answer. In those few instances when we were able 
to contact the client organization, messages were left to the organization's POC, yet 
no return calls were ever received. Some of the client organization POCs who were 
successfully contacted indicated that they never received the survey package. In one 
specific case, the survey package had been addressed to the client organization 
headquarters rather than to the field activity where the organization's POC was 
located. Among the non-respondents contacted, we accounted for three clients who 
did not receive the survey package. Two survey packages were returned for reasons 


of insufficient address or unknown addressee. And lastly, one completed 


questionnaire arrived four weeks after our analysis of the initial set of twenty-six 
responses. Our net sample size after adjustment, therefore, was forty-six which 
yielded a response rate of fifty-seven percent. 

Due to time and schedule constraints, subsequent mailings were not 
conducted. Moreover, we determined that the response rate of fifty-seven percent 
was acceptable for purposes of our evaluation. 

3. Processing of Pilot Data 

The twenty-six returned questionnaires were processed in the following 
manner. As survey packages were returned and after each envelope was opened, the 
questionnaire was reviewed and checked against the master mailing list. Each 
completed questionnaire was then assigned a unique client code. The convention 
used in assigning the client code was straightforward - a number that represented the 
sequence in which the questionnaire was received. The scored results of each 
questionnaire were then entered into a microcomputer based spreadsheet (EXCEL) 
and saved on diskettes for subsequent processing by various statistical software 
programs. The specific software programs and the results obtained from the 


processing of the scores are discussed in the next following sections. 


B. DESCRIPTIVE STATISTICS 

Basic descriptive statistics were used to evaluate the data collected from the pilot 
survey.! Frequency distributions were performed on each item in order to uncover 
any irregularities in the responses. What was found, was an unusually large number 


of not applicable (N/A) responses had been given for many of the items. Figure 4.1 


1 The descriptive statistics developed from the pilot survey are proprietary 
information of NARDAC San Francisco, and will not be disclosed in this thesis. 
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shows a histogram of all the items and their associated percent of not applicable 
responses. A three-step process was used to determine the cause of the high N/A 
response rate. 

The first step was to isolate those items that had an unusually large number of 
not applicable responses; the mean was chosen as the break-point. Any item which 
had a not applicable response rate above 31.4 percent was coded with an asterisk on 
the histogram as shown in Figure 4.1 (i.e., items 5, 6, 9, 17, 18, 19, 20, 21, 22, 
23, 25, 26, 27, 29, 38, and 42). Table 4.1 shows the break-down of the isolated 
items by factor. 

It was interesting to note that all of the items for the factor Quality of Products 

and Services, and three out of the five items for the factor Timeliness of Service, fell 
above the mean (31.4%) N/A response rate. Problems discovered in item 
construction for these two factors will be discussed in Section D of this chapter. 
The next step was to see if there were any relationships between the sixteen items 
listed in Table 4.1 and the respondents which contributed significantly to the N/A 
response rate for those items. Figure 4.2 shows the number of N/A responses for 
the sixteen isolated items by respondent. Again, the mean was chosen as the break- 
point to separate those respondents that had a significantly large number of N/A 
responses. The asterisks in Figure 4.2 indicate which respondents fell above the 
mean (9) rate (i.e., respondents 1, 2, 3, 4, 5, 7, 8, 11, 12, 13, 21, 22, 23, 24, and 
25). 

During the last step, we looked at the profile of the respondents isolated in the 
previous step to determine what may have caused these respondents to answer N/A to 
so many items. Table 4.2 lists all the pilot respondents (by code), NARDAC San 


Francisco's fiscal-year-to-date (FYTD) revenue for that respondent, and the type of 
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TABLE 4.2 Pilot Client Profile Data 


CLIENT CODE 


O 0 Y QA MN A Uy Wo mu 


FYTD REVENUE ($000) 
23 


Al 


SERVICE PROVIDED 


Risk Assessment 

Training 

Study 

Programming 

Training & Procurement 

Training 

Training 

IRC Support/Sperry 

Labor 

Study & Procurement 

Programming 

ADP Security Risk 
Assesment 

Programming 

IRC Support, Procurement, Tech 
Support 

Programming, Telecom, Key- 
Entry, Burroughs, IV Phase 

FYPR-Sperry 

Sperry, Comlines, Terminal 

Sperry 

Labor, Misc, Telecom 

Procurement, Training 

IRC Support, Procurement 

Programming, Procurement 

Training 


IRC Support, Procurement 


For all respondents 
(Except for respondents 13 
& 23) 





service provided. The respondents in bold type are the ones isolated in step two. 


The following observations were made: 


+ The mean FYTD Revenue for all the isolated respondents (except respondents 
13 and 23)! was $7,700. As compared to the total mean for all respondents, 
these are relatively small clients. 


e Forty percent of the highlighted respondents only received training as a 
service. The questionnaire does not accommodate training services as well as 
other services provided by the NARDAC. This problematic area will be 
addressed in Section D of this chapter. It should also be noted that many of the 
NARDACs distribute separate surveys for clients receiving training. 


e The contract for respondent 3 had not begun at the time the pilot was 
conducted. 


+ The contracts for respondents 5 and 25 have been cancelled. 


TABLE 4.1 Break-down of Isolated Factors. 
Accessibility 5 
System Resources 6,9 


Cost Management - 


Quality of Products and Services 17, 18, 19, 20, 21, 22, 23 
Timeliness of Services 25, 20827 


Responsiveness 29 
Staffing 38 


General Business Practices 43 





I Respondent 13 had a one time contract for ADP Security Risk Assessment and had 
little interaction with NARDAC San Francisco. Respondent 23 had a large hardware 
procurement which accounted for 75% of his total FYTD revenue. It was felt that these 
two respondents' relatively high FYTD revenue was not the norm and were disregarded in 
the computation of the mean for the isolated respondents. 
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It would be premature to make any widespread conclusions about the quality of 
the questionnaire from this phase of the evaluation since the number of respondents 
(26) from the pilot survey was small. Further testing of the instrument with a larger 
sample size will be required in order properly evaluate the questionnaire; however, it 
is clear that there are problems with some of the items in the factors Quality of 
Products and Services and Timeliness of Services. These issues will be addressed in 
Section D. 

Generally, the comments about the questionnaire were quite favorable. Only 
seven out of the twenty-six respondents actually commented on the construction of 
the questionnaire. Of those who commented, over seventy percent thought the 
questionnaire was well prepared. A typical comment was, "Your questionnaire was 
quite thorough, easy to read, and concise...the instructions were fine...”. Only one 
respondent (#12) felt the questionnaire was too long, but he is a new client and 
answered thirty-one of the forty-four items on the questionnaire with a not applicable 


response. 


C. VALIDATION AND RELIABILITY PROCEDURES 
The next step in the evaluation of the instrument was to test the questionnaire for 
its validity and reliability. In this study, we examined content validity or face validity 
and construct validity. Cronbach's alpha was used to examine reliability. 
= 1. Content Validation 
Content validation is determined from the content and operations of the 
measurement instrument [Ref. 15: p. 156}. Others, such as Babbie, refer to content 
validity as face validity or logical validity. For instance, if we were indeed interested 
in measuring client satisfaction of those clients supported by the NARDACs, then 


from a logical point of view each of the items considered should appear on its face to 
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indicate the respondent's satisfaction with the services provided by the NARDACs. 
[Ref. 23: p. 256] The content of the measurement instrument, therefore, has to be 
designed to include all important items or attributes affecting a client's satisfaction 
with the NARDAC support. As described in Chapter III, this has been the 
underlying theme of our methodology to identify those items and to test the list of 
items for completeness. 

The items were examined, prior to the pilot, by independent groups of 
experts as the initial phase in validating the completeness of the items list. This 
follows a similar approach used by Pearson in which independent assessments and 
the collective experience of expert groups provided different perspectives to 
counterbalance any significant omissions that could have occurred. [Ref. 16: p.89] 
The first group of experts was selected on the basis of their experience in ADP and 
interaction with the clients. This group was composed of all the NARDAC Client 
Liaison Officers who reviewed the initial draft of the questionnaire. The same 
questionnaire was reviewed by another group of experts in the academic environment 
with research and consulting experience in the fields of data processing, survey 
research and statistics, and organizational management. This second group was 
comprised of faculty in the Information Systems, Operations Research and 
Administrative Sciences curriculums at the Naval Postgraduate School. Based on the 
comments and recommendations from the first two groups of experts, the item list 
was modified and the questionnaire revised. The revised questionnaire was then 
reviewed by a third group of experts with extensive experience in the data processing 
field and management in the Navy and with a history of formal interaction with client 
organizations. This last independent body of reviewers was made up of the 


Commanding Officers of the NARDACs and the NAVDAC staff, in essence, the 


NARDAC Board of Directors. Details of these reviews are described in Chapter III, 
Section D, Parts 3 and 5. The modifications recommended by the three review 
groups and the independent assessment of the completeness of the item list indicated 
that the important items influencing a client's satisfaction with NARDAC services 
had been identified and validated. 

The other aspect of content validity deals with the logical soundness of the 
operations for measuring the content of the items. To establish the operations for the 
measurement of this content, the Likert Scale methodology was used. The selection 
of this methodology for appropriateness and the development of the measurement 
instrument is described in Chapter III. On the basis of this approach, we determined 
that this measurement process was further evidence of content validity, although 
subjective in nature. 

Content validity was likewise performed by the clients themselves during 
the pilot phase of the measurement instrument. The clients participating in the pilot 
were asked to critique the questionnaire by providing an assessment of the following: 

e Clarity of item phraseology 

e Appearance of item redundancy 

e Omission of critical items 

e Readability of the survey and ease of completion 

e Clarity of the instructions 
Their assessment of the above items addressed the questionnaire's content structure 
and completeness of items; therefore, further evidence of content validity had been 


achieved. 
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2. Construct Validation 

In the previous section, content validity of the measurement instrument and 
the process in performing this test for validity was described. Construct validity 
focuses on the nature of the items being measured and the extent to which these 
particular items relate to one another. Thus, it attempts to measure the correlations 
among many independent items to determine whether these items are strongly enough 
related to describe a particular relational concept. In this case, construct validation 
was used to determine how strongly each of the forty-four items related to one 
another and to each of the eight factors. Factor analysis was the technique employed 
in an effort to perform this type of validation. 

a. Factor Analysis Description 

Factor analysis is a multivariate statistical tool that analyzes 
interrelationships among many items (e.g., questionnaire responses) and then 
explaining these items in terms of their common underlying dimensions (factors) 
[Ref. 31: p. 427]. It is, in effect, a simultaneously processed item interdependence 
technique that tells which item responses measure the same factor and to what extent 
they measure these factors. 

In examining the pilot responses for construct validity, we performed 
factor analysis on the forty-four item scores for each of the twenty-six respondents. 
This was accomplished using the factor analysis software module in 
STATGRAPHICS on a Zenith 248 microcomputer and the SPSS-X factor analysis 
program on an IBM 3033/4381 computer. The purpose of these trials was to 
determine if there was indeed some logical pattern among the forty-four items 
intercorrelations and to see what dimensional factors would be generated based on 


the intercorrelations of these empirical input. STATGRAPHICS allows the use of 


the original item responses and prompts for the number of factors to extract in the 
analysis. SPSS-X performs similarly and, as an addition, provides the feature of 
generating the number of factors automatically. 
b. Factor Analysis Procedures 

The forty-four item scores for each of the twenty-six respondents were 
entered into a file for processing by the statistical software programs. In the case of 
STATGRAPHICS, these entries were simultaneously stored by the software 
program to the work diskette. Parameter specifications used for the factor analysis 
runs were Pairwise (to handle N/A items) and Varimax rotation. The output in 
Pearson Coefficient format (available on SPSS-X only) contained more meaningful 
information for evaluation. The interested reader is referred to the 
STATGRAPHICS User Guide [Ref. 32] and SPSS User Manual [Ref. 34] for more 
details on the various factor analysis parameter options. 

c. Interpretation of Factor Analysis Output 

The output of a factor analysis program is presented in matrix form and 
consists of several key components. The columns represent the factors (artificial 
dimensions) generated from the observed relations among items. The values under 
each factor column represent the correlations between each item and each factor and 
is referred to as the factor loadings. In examining the output, one can determine the 
meaning of a given factor based on those items that load highly on it. Babbie points 
out that the generation of factors, however, has no reference to the meaning of the 
item, only their empirical associations. Furthermore, he offers two important criteria 


to consider while evaluating this data. 
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e A factor must explain a relatively large proportion of the variance found in the 
items, and 


+ Every factor should be more or less independent of every other factor. [Ref. 
23: p. 328] 


Although the interpretation of the complex interrelationships found in 
the factor analysis matrix output is no simple matter, the following provides a step- 


by-step method for evaluating the data.! 


1. Examine the factor matrix. Each column denotes a separate factor. The values 
beneath the columns are the factor loadings for each item on each factor. The 
numbers on the left margin of the matrix represent each of the 44 items in the 
questionnaire. 


2. Begin the analysis by starting at the first item on the first factor and move 
horizontally from left to right, looking for the highest loading for that item on 
any factor. For sample sizes less than 100, the lowest factor loading to be 
considered significant would be + .50. If the highest loading is significant, 
underline it. 


3. Proceed to the second item and, again moving from left to right horizontally, 
look for the highest loading for that item on any factor and underline it. 
Continue the procedure for each item until all the items have been underlined 
once for their highest significant loading on a factor. Some items may have 
several loadings. 


4. Identify items that have not been underlined (those that do not load on a factor). 
If the item is considered important, leave the item as is. If the item(s) are 
considered of minor importance to client satisfaction, the item(s) may be 
eliminated and derive another factor analysis solution with the non-loading 
items eliminated. 


5. When all significant items are loading on a factor indicating that a factor 
solution has been obtained, assign a name or meaning to the pattern of factor 
loadings. Items with higher loadings are considered more important. The final 
result will be a label or a name that represents each of the derived factors. [Ref. 
34: pp. 250-251] 


l Itis recommended that the expertise of a social researcher be used in interpreting the 
factor analysis output. 


d. Pilot Results 

The factor analysis execution runs of the pilot data using both 
STATGRAPHICS and SPSS-X produced unusable output and therefore 
inconclusive results. One explanation for these results could be the small sample size 
to item ratio for this pilot (.59 : 1). Hair et. al. emphasizes that, as a general rule, 
there should be five times as many responses as there are items (5:1) to be analyzed 
and that this ratio is considered to be somewhat conservative. He further adds that 
when dealing with smaller sample sizes and a lower sample size to item ratio, any 
findings should be interpreted with caution. [Ref. 34: pp. 250-251] Ideally then, a 
preferred sample size to ensure a more meaningful and empirically sound evaluation 

of the questionnaire should be at least 220 (1.e., 5 times 44 items). 

3. Reliability Testing 

According to Kerlinger, "reliability can be defined as the relative absence of 
errors of measurement in a measuring instrument". In other words it is the accuracy 
or precision of a measuring instrument. [Ref. 30: p. 405] There are basically two 
types of reliability tests that can be performed: test-retest method, and the internal 
consistency method (Cronbach's alpha). According to Carmines and Zeller, test- 
retest method is one of the easiest ways to estimate the reliability of empirical 
measurements [Ref. 31: p. 37]. In this method, the same survey is given to the same 
set of respondents after a period of time, and the correlation between the scores is 
obtained. If exactly the same results are obtained on the two administrations of the 
survey, the test-retest reliability coefficient will be 1.00 (i.e., perfect reliability). The 
problem with this method is that the respondent's perception of client satisfaction 
will presumedly change over time based on the quality of services provided by the 


NARDAC. Thus, a low test-retest reliability correlation may not indicate that the 
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reliability of the questionnaire is low. The test-retest method can also be expensive 
and impractical to administer. 

A much better method of determining reliability is the measurement of the 
amount of error in the instrument. The amount of error in a measure can be 
determined using Cronbach's alpha test applied to interitem correlations, which can 
be expressed as follows: 

Np' 
a = Åm o where 


[1 + ONE 


the number of items 


p 


the mean interitem correlations. 


The value @ will vary between .00 and 1.00, when the mean interitem 


correlations! are between zero and one. The value o at 1.00 is perfect reliability, but 


a reliability score of .80 is considered acceptable for basic research [Ref. 15: p.788]. 
As with the factor analysis procedure, the reliability test was performed, but because 
of the small sample size, the results were inconclusive. It is recommended, however 
that a reliability test be performed on the instrument prior to implementing the Client 
Satisfaction survey. The ratio of sample size to number of items should be at least 
(5:1) or greater. Reliability testing is not available in the STATGRAPHICS program 
however; it can be easily tested using procedure RELIABILITY on SPSS-X. The 
Alpha model (Cronbach's alpha) is the default model in procedure RELIABILITY. 


1 To find the mean interitem correlation, the correlation coefficients are summed and 
divided by the total number of coefficients. 


D. REVISED OUESTIONNAIRE 

The freguency distributions calculated in section B above highlighted two factors 
that displayed an usually high rate of not applicable responses: Ouality of Products 
and Services and Timeliness of Services. Each item within these two factors were 
carefully examined and compared to the profile of the respondents which answered 
not applicable to those items. From our analysis, we concluded that clients, such as 
those receiving training or Information Resource Center (IRC)! assistance, had 
difficulty answering the questions in those two factors. 

As a result, questions 18 and 28 were identified as being too specific and were 
revised to broaden their applicability. Figure 4.3 illustrates the changes made. Of 
course, further testing of the instrument with a larger sample size will have to be 
performed to further refine the instrument, but we feel these preliminary changes wil 
reduce the number of not applicable responses for the factors, Quality of Products 
and Services and Timeliness of Services. See Appendix K for the revised 


questionnaire. 


1 The IRC at NARDAC San Francisco was created to help Navy commands and 
other government agencies to deal with the microcomputer revolution. Basically, it helps 
the client take full advantage of the productivity enhancement capabilities of a 
microcomputer. 
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OUALITY OF PRODUCT AND SERVICES 
(NARDAC's ability to deliver reliable services and excellent products) 


CURRENT: 18. The quality of NARDAC's data processing services, 


REVISED: 18. The quality of NARDAC's services. 
(Services include training, Information Resource Center (IRC) 
assistance, data processing, Client Liaison services, studies, etc. 


TIMELINESS OF SERVICES 
(NARDAC's ability to be punctual and "schedule conscientious" with its products 
and services) 


CURRENT: 28. Timeliness of deliverables from NARDAC. 
REVISED: 28. Timeliness of services from NARDAC. 


(Services include training, Information Resource Center (IRC) 
assistance, data processing, Client Liaison services, studies, etc. 





Figure 4.3 Changes to items 18 & 28. 
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V. DATA ANALYSIS TECHNIOUES 


A. INDEX CONSTRUCTION 


An index is a composite measure very freguently used in social research [Ref. 


27: p. 361]. Babbie lists the following advantages of an index: 


e Allows a researcher to develop a composite measure of variables. 


e A single data item might not have enough categories to provide the desired 
range of variation, but an index formed from several items would. 


e Indexes are efficient data reduction devices : several indicators may be 
summarized in a single numeric score. 


An Index is constructed by accumulating scores assigned to individual attributes. 
The data from the client satisfaction questionnaire was formulated into an index, 
which would empirically describe the level of client satisfaction. The Likert scale 
was quantified by assigning values 5, 4, 3, 2, and 1 to extremely satisfied, satisfied, 
neutral, dissatisfied, and extremely dissatisfied respectfully; the not applicable scores 
were disregarded in constructing the index. According to Babbie, "the Likert format 
lends itself to a rather straightforward method of index construction. Since identical 
response categories are used for several items intended to measure a given variable, 
each such item can be scored in a uniform manner.’ [Ref. 27: p. 375] Using the 
values above, the scores on the questionnaire were averaged for each factor to arrive 


at a factor index: 
IM: 
l 
Fj = F 2, Ijk where 


Fj = individual factor index 
mj = total number of items for factor] 
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L = number of applicable responses 
I;; = numeric score for item k of factor j, 
= 1, 2, 3, 4, 5. 


To determine the overall Client Satisfaction Index (CSI), all of the factor indices (Ej) 


are averaged: 


where 


N = number of meaningful factors 
(1.e., factors where at least one item was applicable) 
i =the number of factors. 


The result will be a CS7 value between one and five. Each factor was given an equal 
weight because there were no substantiating reasons for differential weighting of the 


factors. An interpretation of the index is shown in Table 5.1. 


TABLE 5.1. 


CSI RANGE VALUES INTERPRETATION 


4.21 to 5.00 Extremely Satisfied 
3.41 to 4.20 Satisfied 


2.61 to 3.40 Neutral 
1.81 to 2.20 Dissatisfied 
1.00 to 1.80 Extremely Dissatisfied 





B. IMPORTANCE-PERFORMANCE GRID 
The Importance-Performance Grid was developed by Robert Alloway, from the 
Sloan School of Management, Massachusetts Institute of Technology, to help 


identify which client satisfaction factors are most important and to allocate resources 


to ensure good performance on those factors (i.e., "prioritizes management's 
attention”). [Ref. 29] 

The last section of the questionnaire is used to collect data which can be used to 
develop an importance-performance grid. The respondent is asked to rate the relative 
importance of each of the eight performance related factors on a scale of one to five. 
These importance scores are then plotted against the average performance scores for 
each factor (F j). After all the factors have been plotted, the grid is arbitrarily divided 
into four quadrants. The placement of the axes is a managerial decision based on 
what senior management feels are achievable standards of success (e.g., see Figure 


5.1). Alloway defines the quadrants as follows: 


The upper-right corner is the relative success quadrant. This implies 
no change is necessary in importance-performance for these criteria. 


The lower-left quadrant is also OK in the sense that lower performance 
on these comparatively unimportant criteria is acceptable. My 
recommendation for these two quadrants, leave them be, might appear 
much too benign until one contemplates the level of managerial 
attention and effort required to improve the lower-right quadrant. 


The lower-right quadrant is the real killer. These criteria have high 
importance but low performance. These are the criteria which ruin a 
DP department's reputation, drive users up the wall, seriously impair 
DP's ability to deliver, and prevent user managers from receiving their 
relevant information. 


The upper-left quadrant should receive declining management 
attention. Any increased efforts to improve performance here are a 
relative waste of resources. Clearly DP management should not steal 
resources from the success quadrant for use in the killer quadrant, 
rather, DP should reallocate from waste and OK to killer. [Ref. 29: 
pp. 15-17] 


Each client will have a unique profile based on his Importance-Performance Grid 


that will focus management's attention on the factors that the client feels requires 
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greater attention by the NARDAC. It will also assist the NARDAC in managing 


scarce resources. 


SUCCESS au FACTORS 


a2 - Accessibility 
- System Resources 


- Cost Management 


- Quality of Product 
and Services 


- Timeliness of Services 


uu 
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= 
O 
LL 
cr 
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- Responsiveness 


- Staffing 


KILLER 
- General Business 


Practices 


IMPORTANCE 





Figure 5.1 Importance-Performance Grid [Ref. 29] 


In the above example (Figure 4.1), the grid clearly illustrates how the NARDAC 
performed on the various factors and the relative importance of each of those 
factors!. Using Alloway's definition of the quadrants as stated above, the NARDAC 
is performing quite well in the area of System Resources (2) and Staffing (7), which 
are viewed as important to the client's mission. Therefore, no change in the resource 


allocation is necessary. Accessibility (1) and Responsiveness (6), however, rated 


1 These scores are fictitious and were chosen for illustrative purposes only. 


very poorly in performance. This rating, on face value, would indicate a need by 
management to focus more attention in those two areas; however, this particular 
client views the two factors as relatively unimportant. Therefore, it would be more 
productive to concentrate on the more important factors, Quality of Products and 

Services (4) and Timeliness of Services (5). These two factors were rated high in 
importance but low in performance and should have management's highest attention. 
According to Alloway, resources should be reallocated in order to bring factors in the 
killer quadrant into the success quadrant. He suggests pulling resources out of the 
waste quadrant. In this area, Cost Management (3) and Timeliness of Services (5) 

rated high in performance but low in importance, therefore efforts to improve 


performance would be wasting scarce resources. 


C. FREQUENCY HISTOGRAM 

Another interesting and informative way to view the data from the questionnaire 
is through the use of frequency histograms (Figure 5.2). Histograms are a very 
good and quick visual tool that can be used to see how all the respondents answered 
a particular item or factor. The frequency histogram fills in the holes where the CSI 
leaves off. To illustrate this point, it is plain to see from Figure 5.2 that the clients 
are quite satisfied with the accessibility of the NARDAC with sixty-five percent of 
the respondents answering satisfied or higher. The histogram could also point out a 
poorly constructed item if there was a very large percentage of not applicable 


responses. 
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FREOUENCY HISTOGRAM FOR ACCESSIBILITY 


N/A = NOT APPLICABLE 
1 = EXTREMELY DISSATISFIED 
2 = DISSATISFIED 
3 = NEUTRAL 
4 = SATISFIED 
| 5 = EXTREMELY SATISFIED 
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Figure 5.2 Example Frequency Histogram for Accessibility 
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VI. CONCLUSIONS AND RECOMMENDATIONS 


A. CONCLUSIONS 

In conclusion, the research objectives of this study were accomplished. Client 
satisfaction was defined within a domain of eight factors made up of forty-four 
attributes (items). An instrument (questionnaire) was developed to measure these 
factors and an index was constructed to derive an empirical measure of client 
satisfaction. 

Through the process of conducting personal interviews with clients, holding 
discussions with NARDAC management, and digesting numerous articles and 
journals relating to customer satisfaction, client satisfaction was adequately defined 
by eight factors. Forty-four items were developed to further define the factors. A 
self-administered mail questionnaire was developed to collect the data. This method 
of data collection was viewed to be the easiest and least costly, as far as time and 
personnel resources are concerned, to implement by the NARDACs. Due to time 
and resource constraints, the survey instrument was piloted at one site, NARDAC 
San Francisco, using their fifty-two clients as the sample. Twenty-six respondents 
returned the questionnaire, which yielded a return rate of fifty-seven percent after 
accounting for undelivered questionnaires. Next, the pilot data was examined using 
descriptive statistics. As aresult of this phase of the questionnaire evaluation, it was 
discovered that two factors exhibited a large number of not applicable responses. To 
alleviate this problem, two items (18 and 28) were revised to broaden their 
applicability. Content validity, or face validity, was achieved by subjecting the 


questionnaire to numerous critiques. The critiques were conducted by the nine 
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NARDAC Client Liaison Officers, faculty members at the Naval Postgraduate 
School, the nine NARDAC Commanding Officers, and by twenty-six client 
organizations during the pilot. Construct validity and reliability tests were also 
performed, but due to the small sample size to item ratio (.59 : 1), the results were 
discounted. Further testing with a larger sample will have to be conducted by the 
NARDACs prior to implementing the survey. Several methods of analyzing the 
survey data were presented in Chapter V. An index was developed to empirically 
represent client satisfaction and a suggested interpretation of the index was provided. 


The index is represented by the following equations: 


= factor index for factor j 

= total number of items for factor j 

= number of applicable responses 

= numeric score for item k of factor j, 
= 1, 2, 3, 4, 5. 


nb å 


CSI = = F; where 
1 


l 


CSI = Client Satisfaction Index 
N = number of meaningful factors 
(i.e., factors where at least one item was applicable) 


i = the number of factors. 


An Importance-Performance grid was presented to help NARDAC management 
identify which client satisfaction factors are most important to the clients and how to 
allocate resources accordingly to increase client satisfaction. Another informative 


way to view the data collected from the survey is to use frequency histograms. The 


histograms can easily identify items that are problematic (i.e., an item with a high not 
applicable response rate). As a result of this review, an item may need to be revised 


or maybe even deleted from the questionnaire. 


B. RECOMMENDATIONS 
In this last section of our study, we provide several strongly recommended 
Strategies for implementing the measuring instrument that has been developed. In 
addition, the results and conclusions of our study indicate several areas that may be 
pursued for further research. 
1. Implementation Strategies 
The following is a list of recommendations for measuring and analyzing 


client satisfaction for the Navy Regional Data Automation Centers: 


e More rigorous and extensive validation of the measurement instrument should 
be conducted. With a larger sample size (1.e., at least 220 respondents), the 
measurement instrument could be fully tested and evaluated for construct 
validity using the factor analysis method and for reliability using Cronbach's 
alpha. 


e The results of this study and any follow-on actions or plans should be 
published for dissemination to the NARDAC clients. This information could 
help to bolster general acceptance of the instrument by the client community. It 
is especially important that the clients who participated in the interview and the 
pilot receive this feedback to show confidence that their comments were used 
constructively. 


e The results of this study should also be provided to all NARDAC employees 
affected by this study. Such information would help to generate positive 
support and interest in this area and foster general acceptance of the 
measurement instrument. 


61 


The measurement instrument should be prepared in booklet form to present a 
more professional appearance. The booklet is bound and not loosely attached 
together. Moreover, this is the norm in industry and professionally prepared 
surveys. 


Responsibility for the maintenance and administration of the measurement 
instrument and its associated database should be assigned to the Client Liaison 
Officer of each NARDAC. This provides centralized administration of client 
related data and the possible incorporation of the client satisfaction index (CST) 
into the NARDAC's client profile. 


The instrument should be initially administered to all NARDAC clients to 
establish a baseline CSI by obtaining 100% participation. From our extensive 
research and lessons learned from the pilot, the questionnaire should be 
delivered to the clients in person by a NARDAC representative (e.g., the Client 
Liaison Officer/staff). This ensures certainty of delivery and provides the 
NARDAC representative the opportunity to update profile information 
regarding the client. The return of the questionnaire may be accomplished by 
mail using pre-addressed envelopes with telephone follow-up, or by collecting 
the questionnaire in person on a predetermined date. To keep the CSI database 
current, the survey should be administered at least biannually and may be 
conducted after periodic Advisory Board or Client Council meetings. 


A cover letter, such as the one in Appendix C, should be attached in front of 
the questionnaire. 


The survey results can easily be processed and maintained on a 
microcomputer. A microcomputer-based spreadsheet software package, such 
as LOTUS 1-2-3, can be used to consolidate the data from the questionnaire 
and compute the client satisfaction index. A spreadsheet package with graphics 
capabilities can produce visually enhanced output of the data (e.g., CSI 
histograms, frequency histograms of respondents, or other graphics relating to 
the respondents). 


The measurement instrument should be used to track client satisfaction over 
time and circumstance. With trend analysis, the effects of technological or 
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policy changes implemented by the NARDACS on client satisfaction could be 
identified and evaluated. 


2. Further Research Areas 


The following is a list of recommendations for further research: 


e The forty-four items of the measurement instrument could be re-examined. 
Further examination of the items could provide insight into the characteristics 
of each item which could further define client satisfaction. This would be 
beneficial in identifying performance elements for the NARDACs. 


+ The use of the measurement instrument could be incorporated into a 
productivity model that would include effectiveness and efficiency measures. 
This would allow for the setting of target goals for each NARDAC and the 
measurement of progress toward those goals. {adapted from Ref. 16: p. 191] 


+ The measurement instrument could be tailored to other ADP service 
organizations requiring the measurement of client satisfaction. 
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APPENDIX A. MAJOR NARDAC SERVICES 


DATA PROCESSING SERVICES 
TIME SHARING 
DEDICATED 
ANALYSIS AND PROGRAMMING SERVICES 
MICROCOMPUTER 
MINICOMPUTER 
MAINFRAME 
MICROCOMPUTER TRAINING 
ADP SECURITY SERVICES 
TRAINING 
RISK ASSESSMENT ANALYSIS 
DATA COMMUNICATIONS PLANNING SERVICES 
ADP EQUIPMENT PROCUREMENT ASSISTANCE SERVICES 


TECHNICAL CONSULTATION SERVICES 
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APPENDIX B. CLIENT ORGANIZATIONS INTERVIEWED 


SAN DIEGO: 


COMMANDER, NAVAL AIR FORCE, U.S. PACIFIC FLEET 
COMMANDER, NAVAL SURFACE FORCE, U.S. PACIFIC FLEET 

FLEET ACCOUNTING AND DISBURSING CENTER, U.S. PACIFIC FLEET 
NAVAL AIR STATION MIRAMAR 

NAVAL AIR STATION NORTH ISLAND 

NAVAL AVIATION DEPOT NORTH ISLAND 

NAVAL SEA SUPPORT CENTER, U.S. PACIFIC FLEET 

NAVAL STATION SAN DIEGO 

PERSONNEL SUPPORT ACTIVITY SAN DIEGO 

PERSONNEL SUPPORT ACTIVITY NAVAL TRAINING CENTER 


MONTEREY: 


NAVAL POSTGRADUATE SCHOOL 


SAN FRANCISCO: 


COMMANDER, NAVAL BASE SAN FRANCISCO 

MILITARY SEALIFT COMMAND, U.S. PACIFIC FLEET 
NAVAL AIR STATION ALAMEDA 

NAVAL AVIATION DEPOT ALAMEDA 

NAVY SUPPLY CENTER OAKLAND 

PERSONNEL SUPPORT ACTIVITY SAN FRANCISCO 

SHIP INTERMEDIATE MAINTENANCE ACTIVITY ALAMEDA 


WASHINGTON, D.C. : 


NAVAL DATA AUTOMATION COMMAND 
NAVY INSPECTOR GENERAL 

CHIEF OF NAVAL PERSONNEL 
COMPTROLLER OF THE NAVY 


PERSONNEL SUPPORT ACTIVITY, NAVAL DISTRICT WASHINGTON 
68 


NORFOLK: 


COMMANDER IN CHIEF, U.S. ATLANTIC FLEET 

FLEET ACCOUNTING AND DISBURSING CENTER, U.S. ATLANTIC FLEET 
NAVAL AIR STATION NORFOLK 

NAVAL AVIATION DEPOT NORFOLK 

NAVAL FACILITIES ENGINEERING COMMAND 
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APPENDIX C. CORRESPONDENCE 


17 August 1987 


Commander 

Naval Interview Headquarters 
U.S. Atlantic Fleet 

Norfolk, VA 23150 


Dear Interviewee, 


In a continuing effort to better serve their customers, the Naval Data Automation Command 
(NAVDAC) and the Navy Regional Data Automation Centers (NARDACs) have 
undertaken steps to improve customer satisfaction. As part of this effort, they have 
requested our assistance in developing an index which will be used as a measure of 
customer satisfaction. We are students at the Naval Postgraduate School in the Computer 
Systems Management curriculum. We are conducting this research in partial fulfillment of 
our Master of Science Degree in Information Systems and in hopes of providing NAVDAC 
and the NARDACs with a meaningful and useful tool. 


To develop the customer satisfaction index, we must first identify those critical factors 
which the customers would use to evaluate the services provided by the NARDAC. As 
discussed in our phone conversation, we will be conducting a personal interview with you 
regarding these customer satisfaction attributes on the «date» of August «time». We expect 
the interview to last approximately one hour. Below, is a representative list of factors 
which we feel relate to customer satisfaction. We would appreciate your views on them 
during the interview, or any others that you think may be pertinent. 


Computer capabilities (mainframe/mini/micro support) 
Ease of obtaining services 

Systems design and programming 

Technical support 

ADP consultation and assistance 

Training 

Cost of ADP services 

Customer support 

Timeliness and quality of products and services provided 


o e . . ọọ O 9 oe 04 


If you have any questions concerning the interview or our schedule, please leave a message 
for us at our curriculum office, Autovon 878-2174/2175 or commercial (408) 646- 
2174/2175, and we will contact you as soon as possible. 

We appreciate your assistance in this effort and look forward to meeting you on the <date>. 


Sincerely, 


Prima A. Morris Robert J. Birdwell 
LCDR, USN LT, SC, USN 


70 


19 October 1987 


Dear Client Liason Officer, 


As part of the continuing effort to better serve their customers, the NAVDAC Board of 
Directors requested our assistance in developing an index which will be used as a measure 
of customer satisfaction. “We are students at the Naval Postgraduate School in the 
Computer Systems Technology curriculum. The research we are conducting in this area is 
in partial fulfillment of our Masters of Science Degree in Information Systems, and will 
hopefully provide NAVDAC and the NARDACSs with a meaningful and useful tool. 


In developing the customer satisfaction index, we have identified those critical factors 
which existing customers would use to evaluate the services provided by the NARDAC. 
We obtained these relevant data through a series of personal interviews from a cross- 
sampling of customers served by NARDAC San Diego, NARDAC San Francisco, 
NARDAC Washington and NARDAC Norfolk. These interviews were conducted in the 
format shown in enclosures (1) and (2). In addition to the interviews, we have also 
conducted comprehensive literature searches, reviewed customer satisfaction 
surveys/questionnaires currently in use by government and commercial organziations, and 
interviewed a number of marketing and client liason representatives in private industry to 
complement our research. 


We value your input and field expeerience in the client liason area. Please review the initial 
draft of the customer satisfaction survey (enclosure (3)) and forward your comments to us 
by 1 November 1987. Enclosed is a pre-addressed, stamped envelope for your 
convenience and the timely delivery of your comments. The return date is critical to our 
research deadline; therefore, we would appreciate your cooperation in helping us meet our 
milestones. 


If you have any questions regarding the survey and/or our research, please leave us a 
message at Autovon 878-2174/2175 or commercial (408)646-2174/2175 and we will return 
your call as soon as possible. 


Sincerely, 


Prima A. Morris, LCDR, USN 


Robert J. Birdwell, LT, SC, USN 
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13 November 1987 


Commanding Officer 
NARDAC 


Dear Captain, 

We are forwarding two (2) drafts (revised) of the NARDAC Client Satisfaction Survey, 
which we developed in response to requirements from the NARDAC Board of Directors, 
for your review and comments. Both questionnaires are identical in content but differ in 
format. As part of our research, please indicate which questionnaire format (long form or 
short form) you prefer and why. As requested by Commander Taylor, please forward 
your comments via electronic mail to NARDAC Pearl Harbor by 23 November 1987. 

If you have questions regarding the questionnaires and/or our research, we may be reached 
at Autovon 878-2174/2175 or commercial (408) 646-2174/2175. 


Very Respectfully, 


Prima A. Morris, LCDR, USN 


Robert J. Birdwell, LT, SC, USN 
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7 December 1987 


Commander 

Naval Client 

U.S. Pacific Fleet 

San Francisco, CA 94130 


Dear Client, 


We are students at the Naval Postgraduate School in the Computer Systems Technology 
Curriculum currently developing a Client Satisfaction Index for the Navy Regional Data 
Automation Centers (NARDACs). The Client Satisfaction Index is designed to measure 
each client's level of satisfaction with the products and services provided by the NARDAC. 
In order to arrive at this measurement, a questionnaire is used as the instrument from which 
the Client Satisfaction Index is derived. This questionnaire was developed based on 
comments from NARDAC clients whom we interviewed in your area and other 
geographical locations. 


Please assist us in validating the Client Satisfaction Survey instrument by completing the 
enclosed questionnaire. We also request your personal critique of the questionnaire. 
Please comment on the following: 


. Are there any items you didn't understand? 

. Are there any items you felt were redundant? 

. Åre there any items you feel are critical to client satisfaction that were not 
addressed in the survey? 

. Was the survey easy to read and complete (format)? 

. Were the instructions to complete the survey sufficient? 


WA Di WN — 


Please write your comments directly on the survey below the items you are addressing or at 
the end of the survey. We are soliciting your personal comments, therefore a formal reply 
from your command is not required. Your comments and the results of your completed 
survey will be held in strict confidence, to be used only by us to assess the quality and 
validity of the survey instrument. 


In order for us to meet our scheduled milestones, please return your completed survey in 
the enclosed, pre-addressed envelope as soon as possible but not later than 17 December 
1987. If you have any questions regarding the survey or any aspect of our research 
efforts, we may be reached at Autovon 878-2174/2175 or commercial (408)646- 
2174/2175. 
Thank you for your interest and cooperation. 

Sincerely, 

Prima A. Morris, LCDR, USN 


Robert J. Birdwell, LT, SC, USN 
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<<NARDAC COMMAND LETTERHEAD>> 
Date 
From: Commanding Officer, Navy Regional Data Automation Center, 
To: Commanding Officer, , (Code: XXX) 
Subj: CLIENT SATISFACTION SURVEY 
Encl: (1) Client Satisfaction Survey 


1. As part of a continuing effort to improve our service, please take a few minutes to 
complete the enclosed questionnaire. 


2. The purpose of the Client Satisfaction Survey is to assist us in determining what 
action can be taken to develop more effective ADP support for our customers. Your 
response will enable us to better understand your present and future needs. The 
questionnaire is designed to snapshot present conditions; therefore, please answer the 
questionaire to reflect current conditions. 


3. We feel this survey will be mutually beneficial. We appreciate your continued 
assistance in helping us serve you. 
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APPENDIX D. INTERVIEW INFORMATION SHEET 


DATE OF INTERVIEW: 
NAME OF ORGANIZATION: 
NAME OF INTERVIEWEE: 
GRADE /RANK: 

CURRENT POSITION: 


LENGTH OF TIME 
INVOLVED WITH NARDAC: 


TYPE OF SERVICE(S): 
NARDAC PROVIDES: 


FREQUENCY OF INTERACTION 
WITH NARDAC (i.e., daily weekly): 
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APPENDIX E. INTERVIEW QUESTIONS 


1. What are the factors or areas you feel that the NARDAC customers should use 
to measure customer Satisfaction? 


2. What is, in your opinion, a satisfied NARDAC customer? 


3. Rank the factors in order of importance: 


Computer capabilities (mainframe/mini/micro support) 
Ease of obtaining services 

System design and programming expertise 

Technical support 

ADP consultation and assistance 

Training 

Cost of ADP services 

Customer support 

Timeliness and quality of products and services provided 


4. Specifically, how would you evaluate or measure each of the factors? 
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APPENDIX F. ORIGINAL QUESTIONNAIRE 


1  NARDAC CUSTOMER SATISFACTION SURVEY 


Please place an X in 19 DOX nicnn DE d ibe our level 9 d aciion 


(Additional comments are weicome). 


Customer Support 


1. NARDAC's understanding of the impact on your command for late or inaccurate products O O NOT APPUCABLE 


Of Services. 





2. NARDAC’s familianty with your command’s mission. 


3. How satisfied are you with the lavel of assistance provided in preparing proposals for 


new projects? 


4. Elexibility to satisfactonly respond to changes in your specification requirements. 


5. Courteous and professional attitude of the NARDAC personnel with whom you deal. 











6. Promuigation of general purpose information about what is happening around NAROAC 
(l.e.. now hardware, system software, application software, training, new personnel, etc.) 
through newsietters, electronic bulletin boards, and Client Counci Meetings. 











5 VERY SATISFIED 

4 SATISFIED 

d NEITHER SATISFIED NOR DISSATISF ED 
2 DISSATISFIED 

1 VERY DISSATISFIED 


O o norarrucasie 


5 VERY SATISFED 

4 SATISFIED 

3 NEITHER SATISFIED NOR DISSATISFIED 
2 OSSATISFIED 

1 VERY OSSATISFED 


C o voraPrucaaE 


5 VERY SATISFED 

4 SATISFED 

3 NEITHER SATISFIED NOR DISSATISF ED 
2 DISSATISFED 

1 VERY OSSATSFED 


L] o NOT aPPucaaE 


5 VERY SATISFED 

4 SATISFIED 

d NEITHER SATISFIED NOR OISSATISFED 
2 DISSATISFIED 

1 VERY DISSATISFED 


O o noraprucasie 


5 VERY SATISFED 

4 SATISFED 

3 NEITHER SATISFED NOR DISSATISF ED 
2 DISSATISFED 

1 VERY DISSATISFED 


O o norareucase 


5 VERY SATISFED 

4 SATISFIED 

3 NEITHER SATISFED NOR DISSATISF ED 
2 DISSATISFIED 

Y VERY DISSATISFED 


2 NARDAC CUSTOMER SATISFACTION SURVEY 


7. Timeliness of response for a request for services. 








8. Level of attention given to your project and command, 








9. When a problem associated with a NARDAC product or service is reported, how satisfied 
are you with the timeliness and quality of the resolution? 











Service Level Management, 


10. Accuracy of the intial pilanning/cost estimates for your project request. 





11. Ability to adequately explain costs and verify charges. 











12. Format and quality of the Chargeback Report. 
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O 0 NOT APPLICABLE 
CJ 5 very sansrep 


4 SATSFED 

3 NEITHER SATISFED NORA DISSATISFED 
2 DISSATISFED 

1 VERY DISSATISFED 


O NOT APPLICABLE 
O 5 VERY SATISFED 


4 SATSFED 

3 NEITHER SATISFED NOR DISSATISFED 
2 DISSATISFED 

1 VERY OSSATISFED 


O NOT APPLICABLE 


5 VERY SATISFED 


O e sansren 


3 NEITHER SATISFED NOR DISSATISF ED 
2 DISSATISFED 
1 VERY OSSATISFED 


O NOT APPUCABLE 
O 5 VERY SATISFED 


4 SATISFED 

3 NEITHER SATISFED NOR DISSATISFED 
2 DISSATISFED 

1 VERY DISSATISFED 


O NOT APPUCABLE 
O 5 VERY SATISFED 


4 SATISFED 

3 NEITHER SATISFIED NOR DISSA TISFIED 
2 DISSATISFIED 

1 VERY DISSATISFIED 


O NOT APPLICABLE 
O s verv sansrep 


4 SATISFED 

3 NEITHER SATISFED NOR DISSATISFED 
2 DISSATISFED 

1 VERY DISSATISFED 


3 NARDAC CUSTOMER SATISFACTION SURVEY 


13. 


Current rates NARDAC charges for services are competitive with industry. 











. How satisfied are you with the coordination and follow-up on tasks/projects assigned to 


subcontractors? 





Assessibility 


15. 


16. 


17. 


18. 


Communication channels between you and your NARDAC point of contact. 








Procedures for requesting varous NARDAC services. 











| 











Physical Jocation of NARDAC. 
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GO O NOT APPUCABLE 
U s mary AGREE 


4 AGREE 


g 3 NEITHER AGREE NOR DISAGREE 


2 DISAGREE 
1 HIGHLY DISAGREE 


O NOT APPLICABLE 
O S VERY SATISFIED 


4 SATSFED 

3 NEITHER SATISFIED NOA DISSATISFED 
2 DISSATISFIED 

1 VERY DISSATISHED 


O NOT APPLICABLE 
O S VERY SATISFED 


4 SATISFED 

3 NEITHER SATISFIED NOR DISSATISFIED 
2 DISSATISFED 

1 vERY DISSATISFED 


O NOT APPLICABLE 
O S VERY SATISFED 


4 SATISFED 


O 3 NEITHER SATISFIED NOR DISSATISFED 


2 DISSATISFIED 
1 VERY DISSATISFED 


O NOT APPUCABLE 
A S VERY SATISHED 


4 SATISFED 

3 NEITHER SATISFIED NOR DISSATISFED 
2 DISSATISFED 

1 VERY DISSATISFED 


O NOT APPUCABLE 
O S VERY SATISFED 


4 SATSFED 

3 NEITHER SATISFIED NOR DISSATISFIED 
2 DISSATISFED 

1 VERY DISSATISFED 


å NARDAC CUSTOMER SATISFACTION SURVEY 


19. Vistor parking spaces at NARDAC. 


Hardware, Software, and Communication Technology, 


20. Current computer hardware capabilities (mainframe, minicomputers, microcomputers). 


21. Ability to meet your projected automated information system requirements. 


22. Currency and variety of sottware packages available. 


23. Jelecommunication capabilities and services satisty/meet my command's operational 


needs. 


Technical Proficiency, 
24. Laval of expertise of the technical staff. 


O O NOT APPLICABLE 
S VERY SATISFED 
4 SATSFED 
3 NEITHER SATISFED NOR DISSATISF ED 
2 DISSATISFIED 
1 VERY DISSATISFIED 


O O NOT APPUCABLE 
CJ 5 very sansrep 
4 SATSFED 
3 NEITHER SATISFED NOR DISSATISFED 
2 DISSATISFED 
1 VERY OSSATISFED 


O 0 NOT APPLICABLE 
S VERY SATISFED 
4 SATSFED 
3 NEITHER SATISFIED NOR DISSATISFED 
2 DISSATISFIED 
1 VERY DISSATISFED 


[ul O NOT APPLICABLE 
C] 5 very sansrep 
4 SATISFED 
3 NEITHER SATISFED NOR DISSATISFED 
2 DISSATISFIED 
1 VERY DISSATISFED 


O O NOT APPLICABLE 
S HIGHLY AGREE 
4 AGREE 
3 NEITHER AGREE NOR DISAGREE 
2 DISAGREE 
1 HIGHLY DISAGREE 


O O NOT APPLICABLE 
O 5 VERY SATISRED 
4 SATSFED 
O 3 NEITHER SATISFIED NOR DISSATISFED 
2 DISSATISFIED 
1 VERY DISSATISFED 


S NARDAC CUSTOMER SATISFACTION SURVEY 


25. The technical staff is well versed in a broad range of ADP subject areas. 


26. 


el: 


28. 


29. 











The technical staff is keeping abreast of the latest developments in the ADP arena. 








Ability to design and develop creative and innovative systems. 











Personnel assigned to your project demonstrated the proper skill jaye) required to 
pertorm the task(s) specitied in the Statement of Work. 








Ability of the technical staff to communicate in customer terms/anquage. 








Quality of Products and Services, 
30. The guality and accuracy of deliverables. 
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C] 0 nor aPPUCABLE 


S HIGHLY AGREE 
4 AGREE 


E 3 NEITHER AGREE NOR DISAGREE 


2 DISAGREE 
1 HIGHLY DISAGREE 


O NOT APPLICABLE 
O 5 HIGHLY AGREE 


4 AGREE 


O] 3 NEITHER AGREE NOR DISAGREE 


2 DISAGREE 
I HIGHLY DISAGREE 


O NOT APPUCABLE 
g S VERY SATISFED 


4 SATISFED 


O 3 NEITHER SATISFIED NOR DISSATISF ED 


2 DISSATISFED 
1 VERY DISSATISFIED 


O NOT APPLICABLE 
DO) 5 VERY SATSFED 


4 SATISFED 

3 NEITHER SATISFED NOR DISSATISFED 
2 DISSATISFIED 

1 VERY OSSATISFED 


O NOT APPLICABLE 
O 5 VERY SATISFED 


å SATISFED 

3 NEITHER SATISFED NOR DISSATISF ED 
2 DISSATISFED 

1 VERY OSSATISFED 


O NOT APPLICABLE 
= 5 VERY SATISFED 


4 SATISFED 

3 NEITHER SATISFED NOR DISSATISFED 
2 DISSATISFIED 

I VERY DISSATISFED 


6 NARDAC CUSTOMER SATISFACTION SURVEY 


31. 


32. 


33. 


34. 


35. 


36. 


Availability of online and batch systems (i.e., the total time the system is up). 











Quality of system documentaton/manuals (@.g., readable, correct, up-to-date). 











The performance of your system(s) currently running on NARDAC computers. 

















Response time of online systems. 











Maintenance performed on currem systems (i@., responsiveness in incorporating 
modifications and corrections). 
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O O NOT APPLICABLE 


5 VERY SATISFED 

4 SATISFED 

3 NEITHER SATISFED NOR DISSATISF ED 
2 OISSATISFED 

1 VERY DISSATISFED 


O O NOT APPUCABLE 


5 VERY SATISFED 
4 SATISFED 


O 3 NEITHER SATISFIED NOR DISSATISF ED 


2 DISSATISFED 
1 VERY DISSATISFIED 


O O NOT APPUCABLE 
O 5 very sansrep 


4 SATISFED 

3 NEITHER SATISFED NOR DISSATISFED 
2 DISSATISFIED 

1 VERY DISSATISFIED 


O O NOT APPUCABLE 


S HIGHLY AGREE 
4 AGREE 
3 NETTHER AGREE NOR DISAGREE 


O 2 osacaez 
[] 1 GHLY DISAGREE 


O O NOT APPUCABLE 
O 5 VERY SATISFED 


4 SATISFED 


a 3 NEITHER SATISFIED NOR DISSATISFIED 


2 DISSATISFIED 
1 VERY DISSATISFIED 


O O NOT APPLICABLE 
Ú s very sansrep 


4 SATSFED 

3 NEITHER SATISFIED NOR DISSATISFED 
2 DISSATISFED 

1 VERY OSSATISFED 


7 NARDAC CUSTOMER SATISFACTION SURVEY 


37. Quality of training. 





Timeliness of Products and Services, 


38. Adherance to project scheduias as specified in the Statement of Work. 


39. Adherance to production processing schedules. 


40. Timeliness of progress reports. 


41. Timely notification of any delays in output. 


42. Reliable and timely delivery of output. 


(Qptonal): Name 





Co vor aura 
O 5 venv sansreD 


4 SATISFIED 

3 NEITHER SATISFIED NOR DISSATISF ED 
2 DISSATISFIED 

1 VERY DISSATISFED 


C] o NorAPRucaaLE 
C 5 VERY SATISFED 


4 SATISFED 

3 NEITHER SATISFED NOR DISSATISF ED 
2 DISSATISFED 

1 VERY DISSATISFED 


O O NOT APPUCABLE 


5 VERY SATISFED 

4 SATSFED 

3 NEITHER SATISFED NOR DISSATISFED 
2 DISSATISFED 

1 VERY DISSATISFED 


E O NOT APPUCABLE 


5 VERY SATISFIED 

4 SATSFED 

3 NEITHER SATISFED NOR DISSATISF ED 
2 DISSATISFED 

1 VERY DISSATISFED 


O o norarpucasie 


5 HIGHLY AGREE 
4 AGREE 


C) 3 NEITHER AGREE NOR DISAGREE 


2 DISAGREE 
1 HIGHLY DISAGREE 


O O NOT APPUCABLE 


5 HIGHLY AGREE 

4 AGREE 

3 NEITHER AGREE NOR DISAGREE 
2 DISAGPEE 

1 HIGHLY DISAGREE 





Thank you for completing this survey. The enciosed envelope has been provided for your convenience. 


APPENDIX G. QUESTIONNAIRE SECOND DRAFT 
SHORT FORM 


NAROAC CLIENT SATISFACTION SURVEY 


PLEASE PLACE AN X IN THE BOX WHICH BEST DESCRIBES YOUR LEVEL OF SATISFACTION WITH EACH OF THE 
FOLLOWING ITEMS. (Additional comments may be made on the last page). 


USE THE FOLLOWING SCALE: 


NA NOT APPLICABLE 5 VERY SATISFIED 


4 SATISFED 3 NEITHER SATISF ED NOR DISSATISFIED 2 CISSATISFIED 


1 VERY OISSATISFIED 





Accessibility, 


de 


How satistied are you wrth the availability of your NARDAC point of contact. 


The groceduras for obtaining vanous products and services. 
(refer to NAVDACINST 5230.1) 


NARDAC’s current hours of operation. 


The ghysical location of NARDAC relative to your command. 


The quantity of ystor parking spaces at NARDAC. 


system Resources, 
(Encompasses hardware, software, and teleprocessing technology) 


6. 


How satisfied are you wthNARDAC's current computer hardware capabilities, 


(¡,0., mauntramo, Mincomputers, MICrOCOMPUIAS) 


NARDAC ability to meet your projected automated intormaton system requirements. 


The currency and vanety of software packages available. 


NARDAC's ability to provide taleormcessing services to meet your 


command’s operational needs. 


Cost Management, 


How satisfied are you with NARDAC's adherence to groigd budaats? 


NARDAC's ability to adequately explain cost and verify charges. 


The format of the Chargeback Report (ie, easy fo read and understand). 


. The accuracy of the Chargeback Report 


The current rates NARDAC charges for products and services... 


wal] 
wal] 
wal] 
wal] 
wal] 


wal] 


wal] 
wal] 
wal] 


nel 
wal] 
wal] 
wal] 
wal] 


b A A A 
ñ 4 4 Ò À 


a ao de 
ara RO 
A À 


bro a qa E 


A à ù A 
500,644 


ata ta e 


a) 4 
d] OG 
Al d 
Al 0 
O DO 
A O 
A A 
IO 
A Q 
Ad] 0. 
Al oO 
Al 0 
Al 0 
Ad] @ 


NAROAC CLIENT SATISFACTION SURVEY 
USE THE FOLLOWING SCALE: 


N/A NOT APPLICABLE § VERY SATISFIED å SATISFED 3 NEITHER SATISFIED NOR DISSATISFED 2 DISSATISFIED 1 VERY DISSATISFIED 





15. Flexibility in charging for services (e.g.. fived-price contract, tiered rate structure, etc.). NAL] sd) aI] a] 20) DO 


(NARDAC 's abilty to deliver reliable services and excellent products) 


16. How Satisfied are you with the quality of NARDAC software products? wal] EE aL] al) Å] D 

17. The aualty of NARDAC's data processing services, Ta sd] aL] a] A) DO 

18. The controls over data security . Ta d] aL] A] Ad] 0 

19. The reliability of online and batch systems (¡.e., the total time the system is up). wal d] aL] a] A) 0 

20. The accuracy of system documentation/manuais. waL] sd] aL] al] A] 1E 

21. The performance of your system(s) currently running on NARDAC computers. wal] d] aL] A] A] iC) 

22. The response time of NARDAC’s online systems. Ta Å] aL] a] Å] O 

23. When a problem associated with a NARDAC product or service is reorted, how satistied. Ta d] aL] A] A) 0 
are you with the quality of the resolution? 

jimeliness of Services, 

24. How satisfied are you with NARDAC's adherence to proiscischedules as specified in the NAL] sd) aL] al] Ad] 0 
Statement of Work? 

25. Adherence to production processing schedules, Ta d] aL] Ad] Ad] OD 

26. Timeliness of progress reports. Ta sd] aL] Ad] A) i 

27. Software mairtanance performed on existing systems (i.e., responsiveness in NAL] sd] aL] al) Ad] UO 
correcting program errors). 

28. Timeliness natitication of delays in output. wal] Å] «0 a] £] D 

29. Timeliness of deliverables Ta d] a] AI] A] O 

30. Timeliness of response for a request for services. wal] sl] a] al] A] DO 
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NARDAC CLIENT SATISFACTION SURVEY 
USE THE FOLLOWING SCALE: 


N/A NOT APPLICABLE 5 VERY SATISFIED 4 SATISFED J NEITHER SATISFIED NOR DISSATISFED 2 DISSATSFED 1 VERY DISSATISFIED 





31. When a problem associated with a NARDAG product or service is reported, how satisfied na L] a] O A] O 
are you with the umeliness of the resoluton?. 


Competency 

32. How satisfied are you with the technical staff's level of expertse?. wal] L] 'n A] A] iL) 
(Technical sattt includes computer specialists, programmers, and system analysts) 

33. The technical staffs range of knowledge in their respective ADP fieids.. wal] L] 'n a) A] J 

34. The technical staff's knowledge of the latest develooments in their wal] q_] 'n al) A] O 
respective ADP fields. 

35. NARDAC's abilty to design and develop creative and innovative systems. wal] L a] al] A] J 

36. The NARDAC personnel assigned to your project (i.e..do they demonstrate the wal] EN «O a] AI] O 
proper skill level to pertorm the task(s) spectied in the Statement of Work. 

Coordination and Communications 

37. How satistied are you with NARDAC's understanding of the impact on your command for wal] d] aL] a] AI] O 
late of inaccurate products or services? 

38. NARDAC's familianty with your command's mission. wal] EN aL] a] A] O 

39. The level of assistance provided in preparing proposals for new projects NAD d] aL] aL) AI] O 

40. Elexibility to satistactorily respond to changes in your specification requirements. wal] sd] aL] a] a] O 

41. Couneous and professional attitude of the NARDAC personnel with whom you deal. Na] I] aL] a] A] i 

42. Promuiganon of general pumossa information about what is happening around NARDAC Ta -E «0 aL) A] iL) 
(¡.0., now hardware, system software, application software, traming, new personnel, etc.) 
through newsletters, electronic bulletin boards, and Cliem Council meetings. 

43. The coordination and follow-up on tasks/proiects assigned to subcontractors. na] Å] a] al) A] O 

44. The ability of NARDAC's technical staff to communicate in client terms/anguage, wal] dl] DI al) A] 0 


(Technical satff inciudes computer specialists, programmers, and system analysts) 
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NARDAC CLIENT SATISFACTION SURVEY 
USE THE FOLLOWING SCALE: 


NA NOT APPLICABLE 5 VERY SATISFIED 4 SATISFED J NEITHER SATISFIED NOR DISSATISFED 2 DISSATISFIED 1 VERY DISSATISFIED 





PLEASE PLACE AN X IN THE BOX WHICH BEST DESCRIBES THE RELATIVE IMPORTANCE TO YOUR 
ORGANIZATION OF EACH OF THE FOLLOWING PERFORMANCE CRITERIA AS IT RELATES TO CLIENT 
SATISFACTION. 


USE THE FOLLOWING SCALE: 5 CRITICAL 4 IMPORTANT 3 AVERAGE 2 NOT IMPORTANT 1 IRRELEVANT 


1. Accessibility sl] 4} al] o 
o 
DO 
¡0 
DO 
¡0 
o 


2. System Resources 


3. Cost Manegement 


4. Quailty of Producte end Services 


5. Timelineee of Services 


6. Competency 


Heo Gb Å 
Hy ee Gee) å 
h Èb AGG 
O 01 Ge) G 


7 Coordinetlon and Communications 





Commente: 


























¡o A 


f 








PLEASE PAINT 


Thank you for completing this survey. The enclosed envelope has been provided for your convenience. 


APPENDIX H. OUESTIONNAIRE SECOND DRAFT 


NAROAG CLENT SATISFACTION SURVEY 


LONG FORM 


Please place an X in the box which best desenbes your level of sansfaction with each of the following statements: 
(Additional comments are welcome). 


Accessibility 


te 


system Resources, 
(Encompasses hardware, software, and teleprocessing technology) 


6. 


(refer to NAVDACINST 5230.1) 


NARDAC's current hours of operation. 








The procedures for obtaining vanous products and services. 


The ohysical locanon of NARDAC relatrve to your command, 


The quantty of visitor parking spaces at NARDAC. 


How satisfied are you with the availability of your NARDAC point of comact? 


How satisfied are you with NARDAC's current computer hartware capabilities, 
(.@e.,.mantrame, mintcomputers, microcomputers) ? 


(el O NOT APPUCABLE 
5 VERY SATISFED 
4 SATSFED 
3 NEITHER SATISFED NORA DISSATISFEO 
O 2 DISSATISFED 
1 VERY DISSATISFED 


O O NOT APPLICABLE 
O S VERY SATISFIED 
4 SATSFED 
3 NEITHER SATISFIED NOR DISSATISFED 
2 DISSATISFIED 
1 VERY DISSATISFIED 


O o norarmucasiz 
C 5 ver sanse 
4 SATSFED 
3 NEITHER SATISFIED NOR DISSATISFIED 
2 DISSATSFED 
1 VERY OSSATSFED 


O O NOT APPUCABLE 
O 5 VERY SATISFED 
4 SATSFED 
3 NEITHER SATISFED NOR DISSATISFED 
2 DISSATISFED 
1 VERY DISSATISFIED 


O o norarmucase 
O S VERY SATISFIED 
4 SATISFED 
3 NEITHER SATISFED NOR DISSATISFED 
2 DISSANSFED 
1 VERY DISSATISFIED 


O Q NOT APPLICABLE 
O S VERY SATISFED 
4 SATSFED 
3 NEITHER SATISFED NOR DISSATISFED 
2 DISSATISFED 
1 VERY DISSATISFIED 


NARDAC CLIENT SATISFACTION SURVEY 
7. NARDAC's abiity to meet your projecied automated informadon system requirements. 


8. The currency and yanely of sottware packages available. 





9. NARDAG's ability to provide leienrocessing semices to meet your command's 
operational needs. 


Cost Management, 


10. How satisfied are you with NARDAC's adherence to project budgets? 


11. NARDAG's ability to adequately explain costs and verity charges. 


12. The format of the Chargeback Report (/.e., easy to read and understand). 
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C] o Nor APPUCABLE 


5 VERY SATISFED 

4 SATSFED 

3 NEITHER SATISFED NOR DISSATISFED 
2 DISSATISAIED 

1 VERY DISSATISFED 


C o nor aPPUCABLE 
O s very sarsreo 


4 SANSFED 

3 NEITHER SATISFED NOR DISSATISFED 
2 DISSATISFED 

1 VERY DISSATISFED 


O o nor APPLICABLE 


5 VERY SATISFED 

4 SATISHED 

3 NEITHER SATISFED NOR DISSATISFED 
2 DISSATISFED 

1 VERY OSSATISFED 


O o norarrucame 


5 VERY SATISFED 

4 SATSFED 

3 NEITHER SATISFED NOR DISSATISFED 
2 DISSATISFED 

1 VERY DISSATISFIED 


O o vorace 


5 VERY SATISFIED 

4 SATSFED 

3 NEITHER SATISFIED NOR OISSATISFED 
2 DISSATISFED 

1 VERY DISSATISFIED 


O o norarrucame 


5 VERY SATISFED 

4 SATSFED 

3 NEITHER SATISFIED NORA DISSATISFIED 
2 DISSATISFIED 

1 VERY DISSATISFIED 


NARDAC CLIENT SATISFACTION SURVEY 
13. The accuracy of the Chargeback Report a O NOT APPLICABLE 
5 VERY SATISFED 
4 SATSFED 
3 NEITHER SATISFIED NOR DISSATISFED 
2 DISSATISFED 
1 VERY DISSATISFIED 


14. The current rates NARDAC charges for products and services. O PR 
S VERY SATSFED 
4 SATSFED 
3 NEITHER SATISFIED NOR DISSATISFED 
2 DISSATISFED 
1 VERY OSSATISFED 


15. Elexibility in charging for services (e.g., fixed-price contract, iered rate structure, etc.). O QO NOT APPLICABLE 
5 VERY SATISFED 
4 SATISFED 
3 NEITHER SATISFED NOR DISSATISFED 
2 DISSATISFED 
1 VERY DISSATISFED 


Quality of Products and Services, 


16. How satisfied are you with the quality of NARDAC's sofware produais? O O NOT APPUCABLE 
O 5 VERY SATISFED 
å SATISFED 
3 NEITHER SATISFED NOR DISSATISFED 
O 2 DISSATISFED 
1 VERY DISSATISFIED 


17. The quaiity of NARDAC's data processing services. FI O NOT APPLICABLE 
5 VERY SATSFED 


4 SATSFED 

3 NEITHER SATISFIED NOR DISSATISFIED 
2 OSSATISFED 

1 VERY DISSATISFIED 


18. The controls over gala security. O O NOT APPUCABLE 
5 VERY SANFED 


d SATISFED 

3 NEIMHER SATISFED NOR DISSATISFED 
2 DISSATISFED 

1 VERY DISSATISFIED 





NARDAG CLENT SATISFACTION SURVEY 


19. 


20. 


21. 


The taliability of online and batch systems (¡.e., he total time the system is up). 





























. The response time of online systems. 








. When a problem associated with a NARDAC product or service is reported, how satisfied 


are you with the quality of the resolution? 











Iimeliness of Services, 


24. How satisfied are you with NARDAC’s adherence to prmiect schedules as specified in the g O NOT APPUCABLE 


Statement of Work? 








O o norarmucame 
O s very sarsreD 


4 SATSFED 


O 3 NEITHER SATISFIED NOR DISSATISFED 


2 DISSATISFED 
1 VERY DISSATISFIED 


O 0 norarmucanie 
O s very sarsren 


4 SATSFED 

3 NEITHER SATISFIED NOR OISSATISFED 
2 OISSATISFED 

1 VERY DISSATISFED 


O O NOT APPUCABLE 
[] 5 verv anse 


4 SATISFED 

3 NEITHER SATISFIED NOA DISSATISFED 
2 OSSATISFED 

1 VERY DISSANSFED 


Å] o NOTAPRUCABLE 
O 5 verv unse 


4 SATSFED 


O 3 NEITHER SATISFIED NOA DISSATISFED 


2 DISSATISFED 
1 VERY DISSATISFED 


O o norarmucanie 


S VERY SATISFIED 

4 SATSFED 

3 NEITHER SATISFIED NOR DISSATISFIED 
2 OSSATISRED 

1 VERY DISSANSFED 


[] 5 very sansren 


d SATSFED 

3 NEITHER SATISFIED NOR DISSATISFED 
2 DISSATISFED 

1 VERY DISSATISFED 


NARDAC CLENT SATISFACTION SURVEY 


25. Adherence to production processing schedules. O O NOT APPLICABLE 


26. 


28. 


29. 


30. 


5 VERY SATISFIED 

4 SATISFED 

3 NEMHER SATISFIED NOR DISSATISFED 
2 DISSATISFED 

1 VERY DISSATISFIED 


Timeliness of progress reports. O O NOT APPUICABLE 
O 5 VERY SATISFED 


4 SATISFED 

3 NEITHER SATISFIED NOR DISSATISFED 
2 DISSATISFED 

1 VERY DISSATISFIED 


. Sofrware Maimenanca pertormed on existing systems (1.0., rESPONSIVENESS in correcting O O NOT APPLICABLE 


program errors). O 5 VERY SATISFED 
4 SATISHED 


3 NEMHER SATISFED NOR DISSATISFED 
2 DSSATISFED 
1 VERY OSSATISFED 


Timeliness of notificanon of delays in output. O O NOT APPLICABLE 
O S VERY SATISFED 


4 SATSFED 

3 NEITHER SATISFED NOR DISSATISFED 
2 DISSATISFED 

I VERY DISSATISFED 


Timeliness of deliverables, O D NOT APPLICABLE 
5 VERY SATISFED 
4 SATSFED 
3 NETTHER SATISFIED NOR OISSATISFED 
2 DISSATSFED 
1 VERY DISSATISFIED 


Timeliness of response for a request for services. g O NOT APPLICABLE 
5 VERY SATISFED 


4 SATSFED 

3 NEITHER SATISFED NOR DISSATISFED 
2 DISSATISFIED 

1 VERY DISSATISFIED 


NARDAC CLENT SATISFACTION SURVEY 


31. When a problem associated with a NARDAC product or service is repornied, how satisfied O O NOT APPUCABLE 


are you with the Lmeliness of the resolution? is] 5 VERY SATISFED 
Ds SATISFED 


O 3 NEITHER SATISFED NOR DISSATISFED 
2 DISSATISFED 
1 VERY DISSATISFED 











Competency 


32. How satistied are you with the technical staff's leyel of expertisa? O O NOT APPUCABLE 


(Technical staff includes computer specialists, programmers, and sysiem analysts) OD 5 VERY SATISFED 
4 SATSFED 


3 NEITHER SATISFIED NOR DISSATISFED 
2 DISSATISFED 
1 VERY OSSATISFED 








33. The technical statfs range ol knowledge in ther respective ADP fleids. O O NOT APPLICABLE 
5 VERY SATISFED 


4 SATSFED 

3 NEITHER SATISFIED NOR DISSATISFED 
2 DISSATISFED 

1 VERY DISSATISFIED 





34. The technica! staff's knowledge of the latest develooments in their respective ADP . O O NOT APPLICABLE 


fields. O 5 VERY SATISFIED 
d SATISFED 
3 NEITHER SATISFED NOR DISSATISFIED 
2 DISSATSFED 
1 VERY DISSATISFIED 











35. NARDAG's ability to design and develop creative and innovative systems. O O NOT APPUCABLE 
S VERY SATISFED 


4 SATISFED 

3 NEITHER SATISFIED NOR DISSATISFED 
2 DISSATISFED 

1 VERY DISSATISFED 











36. The NARDAC personne! assigned to your project. (i.e., do they demonstrate the E O NOT APPUCABLE 
proper skill leve! to pertorm the task(s) specified in the Statement of Work. 5 VERY SATISFIED 
O 4 SATSFED 
3 NEITHER SATISFIED NOR DISSATISFEED 
2 OSSATISFED 
1 VERY DISSATISFIED 











NARDAC CLIENT SATISFACTION SURVEY 


38. 


39. 


40. 


41. 


42. 








NARDAC's tamihanty with your cammand's mission. 




















Elaubility to satstactonly respond to changes in your specification requirements. 





Courteous and professional attitude of the NARDAC personnel with whom you deal. 








Promuigation of general purpose information about what is happening around NARDAC 











O o NOT APPUCABLE 
[] 5 verv sansren 
O 4 sansrep 


3 NEITHER SATISFED NOR DISSATISFED 
2 DISSATISFED 
1 VERY DISSATISFIED 


O o norareucamie 
O s very sarsreD 


4 SATSFED 

3 NEITHER SATISFIED NOR OISSATISFED 
2 DISSATISFIED 

1 VERY DSSATISFED 


[1 0 NOT APPUCABLE 
[] 5 very sansren 


4 SATISFED 

3 NETTMER SATISFIED NOR DISSATISFIED 
2 DISSATSFED 

1 VERY DISSATISFIED 


O 0 norareucame 


S VERY SATISFED 

4 SATISFED 

3 NEITHER SATISFIED NOR DISSATISFIED 
2 DISSATSFED 

1 VERY DISSATISFIED 


O 0 norarrucane 


5 VERY SATISFIED 

4 SATSFED 

3 NEITHER SATISFIED NOR DISSATISFED 
2 OSSATISFED 

1 VERY DISSATISFED 


O o NOTAPPUCABLE 
C 5 verw anse 


4 SATIBHED 


O 3 NEITHER SATISFED NOR DISSATISFED 


2 DISSATISFIED 
1 VERY DISSATISFIED 


NARDAC CLIENT SATISFACTION SURVEY 
43. The coordinaton and follow-up on tasks/oroiects assigned Jo subcontractors, O O NOT APPLICABLE 
O 5 VERY SATISFED 
4 SATISFED 
3 NETTHER SATISFED NOR DISSATISFIED 
2 OSSATSFED 


1 VERY DISSATISFED 











C) o nor auca 
O 5 VERY SATISFED 
4 SATSFED 
g 3 NEITHER SATISFED NOR DISSATISFED 
2 DISSATISFED 
1 VERY DISSATISFIED 


44. The ability of NARDAC's techrucal staft to communicate in client terms/ianguage. 
(Technical staff includes computer specialists, programmers, and system analysts) 








Please piace an X in the box which best describes the relative mpanance of each of the following performance crotena: 


- 











1. Accessibility O S CAMKCAL 5. Timsiiness of O 5 CRITICAL 
4 MPORTANT Service 4 MPORTANT 
3 AVERAGE O 3 AVERAGE 
O 2 NOT IMPORTANT 2 NOT MPORTANT 
1 IRRELEVANT 1 IRRELEVANT 
2. System Resources O S CRITICAL Compstency O 5 CRITICAL 
O å MPORTANT 4 MPORTANT 
3 AVERAGE 3 AVERAGE 
2 NOT IMPORTANT 2 NOT IMPORTANT 
O 1 IRRELEVANT 1 RELEVANT 
3. Cost Menagement O 5 CRITICAL Coordination and O S CRITICAL 
4 MPORTANT Communications å MPORTANT 
O 3 AVERAGE 3 AVERAGE 
2 NOT MPORTANT 2 NOT MPORTANT 
1 IRRELEVANT 1 RAELEVANT 
4. Quslity of Products O S CRMCAL 
end Services 4 MPORTANT 
O 3 AVERAGE 
O 2 NOT MPORTANT 
1 IRRELEVANT 
Name Cade Telephone, 
PLEASE PAINT 


Thank you for complering this survey. The enclosed envelope has been provided for your convenience. 


APPENDIX I. PILOT POPULATION 


Naval Surface Force,U.S. Pacific Fleet, San Diego 

Fleet Accounting and Disbursing Center U.S. Pacific Fleet, San Diego 
Fleet Hospital Support Office, Alameda 

Naval Station, Treasure Island, San Francisco 

Naval Air Station, Lemoore 

Naval Air Systems Command, Naval Air System Command HQ, Washington, D.C. 
Naval Biosciences Laboratory, Oakland 

Naval Strike Warfare Center, Naval Air Station, Fallon 
Shipbuilding,Conversion and Repair USN, Hunters Point, San Franciscco 
Naval Weapons Station, Concord, CA 

Naval Supply Systems Command, Washington, D.C. 

SIMA NRMF San Francisco, NAS Alameda 

USS Enterprise CVN-65 

Naval Base, San Francisco 

Navy Public Works Center San Francisco Bay 

Mare Island Naval Shipyard, Vallejo 

Naval Education and Training Financial Information Processing Center, NAS 
Pensacola 

Navy Accounting and Finance Center, Washington, D.C 

Naval Air Station, Alameda 

Naval Reserve Force, New Orleans 

Naval Air Station, Moffett Field 

Service Group 1, Oakland 

Shipbuilding, Conversion and Repair USN, Seattle 

U.S. Maritime Defense Zone Pacific, Alameda 

Navy Military Personnel Command, Washington, D.C. 

Naval Aviation Depot, Naval Air Station, Alameda 

Navy Comptroller Standard System Activity, Pensacola 

Naval Supply Center, Oakland 

Personnel Support Activity, San Francisco 

Military Sealift Command Pacific, Oakland 
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Naval School Physical Distribution Management, Oakland 
Naval Medical Command, Northwest Region, Oakland 
Special Boat Unit Eleven, San Francisco 

Naval Postgraduate School, Monterey 

Naval Facilities Engineering Command, Alexandria, VA 
USS Kiska 

Naval Supply Center Puget Sound, Bremerton, WA 

Joint Military Postal Activity Pacific, San Francisco 
Marine Barracks, Naval Air Station, Alameda 

Sacramento Army Depot, Sacramento 

Navy Astronautics Group, Point Mugu, CA 

Chief Preservation Assistance Branch, San Francisco 
Navy Resale and Services Support Office, Field Support Office, Oakland 
Navy Legal Service Office, San Francisco 

Mare Island Naval Shipyard, Vallejo 

Naval Station Mare Island, Vallejo 

Tactical Electronic Warfare Squadron One Twenty Nine, NAS Whidbey Island 
Defense Subsistence Region Pacific, Alameda 

Naval Air Station, Fallon 

Naval and Marine Corps Reserve Center, San Francisco 
Naval Electronic Systems Engineering Center Vallejo 
NARDAC Newport 
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APPENDIX J. PILOT OUESTIONNAIRE 


NAROAC CLIENT SATISFACTIOM SURVEY 


PLEASE PLACE AN X IN THE BOX WHICH BEST DESCRIBES YOUR LEVEL OF SATISFAGTION WITH EACH OF THE 
FOLLOWING ITEMS. (Additional comments may be made on the last page). 


USE THE FOLLOWING SCALE: 


NA NOT APPUCABLE 5 EXTREMELY SATISFED 4 SATISFIED 3 NEUTRAL 2 DISSATISFIED 1 EXTREMELY DISSATISFED 





(Ease of obtaining service) 





1. — The availability of your NARDAG point of contact. wl] di] O OD O 

2. The orocadures required inNAVDACINST 5230.) tor optaining various NARDAG products NAL] O aL] A) Ad] od 
and Services. 

3. The current hours af operation at NARDAG. na O aL] A] A] iC] 

4. The ean el NARDAG reiative to your command. wa] O aL] Q A] iC] 

5. The quantry of visor parking spaces at NARDAC. wal] d] a] a] Ad] O 

system. Resources. 

(Encompasses hardware, software, and teleprocessing technology) 

6. The currert computer hardware capabilities at NARDAC. wa] sl) aL] A] A] O 
(1.0., manframe, MInCompulers, MIEFOCOMPUTOFS) 

7. The ability ot NARDAC to meet your ore: d] £O 4] A] DO 

8. The currency and varaty of software packages available at NARDAG. wal] O DO) I] Å] iC] 

9. The ability ot NARDAC to provide adequate (glenmcessing services to meet your wal] 0 'n Al] A] iC] 
command's operational needs. 

Cost Management, 

(Includes controls over project budgets, rates charged for services, billing procedures, 

and pnce structures) 

10. NARDAC's ability to provide an accurate assessment of both wal] d] n Al] Ad] DO 
the costs and the schedule for a project. 

11. NAROAC’s adherence to groiect budgets wal di] dl] al] A] iC] 

12. The ability of NARDAC to adequately explain orpiect costs ml td] OD oO Oo 

13. The format ot NARDAC's Chargeback Report (Le., easy to read and understand). NAL] sl] 'n A] Ad] 0 

14. The accuracy of NARDAC's Chargeback Report. wal] d] al] 4] A] CJ 
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NARDAC CLIENT SATISFACTION SURVEY 
USE THE FOLLOWING SCALE: 


NA NOT APPLICABLE 5 EXTREMELY SATISFIED 4 SATSFED 3 NEUTRAL 2 DISSATISFIED 1 EXTREMELY OSSATISFIED 





15. The current ratas NARDAC charges for products and services. wal] sl] «O al] a) DO) 


16. Elexibility in charging for NARDAC products and services. wal] d] ad] al] A) 10 


(@.9., fixed-price contract, tiered rate structure, etc.). 


Quality of Products and Services, 
(NARDAC's ability to deiver relabie services and excellent products) 


17. The quality of NARDAC developed application system products. vd] dj Of) D O 

18. The quality ot NARDAC's data orocessing services, wal] sd) «O a A) O 

19. The security controls over your data that is maintained on NARDAC computers. wal] sd] aL] a] AI] DO 
(i.0., Physical security, user «passwords, etc.) 

20. The reliability of NARDAC onime and batch systems (i.e., the total time the system is up). na d] «O al] a] J 

21. The accuracy of NARDAC’s system documentation/manuais. Na] d] aL] A] a] i 

22. The response time of NARDAC's online systems. wal] L] aL] al) A) UO 

23. The quality of the resolution to å problem associated with a NARDAC product or service. na] cE dd] a] a) J 

Timeljness of Services, 

(NARDAC's ability to be punctual and * schedule conscientious* with is products and services) 

24. NARDAC's adherence to project schedules as specified in the NAL] sd] «O Ad] a) J 
Proyect Request. 

25. The ability of NARDAC to adhere to production processing schedules. wal] sd] «O al] A] DO 

26. The trequency of NARDAC progress reports, wal] sd] aL] a] a] DO 

27. The timely notification of delays in output NAT] {£ a] a] A) DO 

28. Timeliness of delivarabies from NARDAC. wal] I D 4] a] 0 

Responsiveness 

(How quickly NARDAC reacts or responds to cient needs) 

29. Software maintenance pertermed on existing NARDAC systems (ie., responsiveness in wal] sd] a] a] A] DO 


correcting program errors). 


| NARDAC CLIENT SATISFACTION SURVEY 
USE THE FOLLOWING SCALE: 


NA NOT APPLICABLE 5 EXTREMELY SATISFIED 4 SATSFED J NEUTRAL 2 DISSATISFIED 1 EXTREMELY DISSATISFIED 





30. The rasponsiveness of NARDAC to a request for services. Na Q aL] al) A) 0 

31. The reponsiveness of NARDAC to resolving a reported problem associated with a product wal] £ n A] A] 0 
or service. 

32. The flexibility of NARDAC to changes in your project requirements. wal] Q O L] a] iL] 

33. The level ot assistanca provided in prepanng pmpasals tor new projects. wal] CE «O aL] 1] O 

Staffing, 

(Encompasses the resourcefuiness and requisite qualifications of NARDAC personne) 

34. The overall range of expertise within the NARDAC organation. na] m a] AL) a] iL] 

35. The level of expertise of the NARDAC personnel with whom you deal. wal] Q «O al) A] iJ 

36. The ability of NARDAC personnel to keep abreast of the igtest developments in their N/A L] a] al) A) iL] 
respective funcnonal area. 

37. The couneous and professional attitude of the NARDAC personnel with whom you deal. wal] sd] a_] al) a] iL) 

38. The technical staff's ability to design and develop creative and innovative systems. wal] L] n a] a] DO 
(Technical staff includes computer specialists, programmers, and system analysts) 

39. The technical staff's ability to satistactority analyze and document your proe na Q dd] al) A) DO 
requirements, 

40. The ability of NARDAC's technical staff to communicate in client terms/anauace. NAL) L a_] al) A] J 

Genera! Business Practices, 

(The manner in which NARDAC conducts business with ts clients) 

41. NARDAG's familianty with your command's mission as it relates to data processing. Ta J] 'n al] AI] LI 

42. NARDAG's understanding ot the impact on your command for late or inaccurate products na] Q a] al) a] LI 
or services. 

43. The coordination and follow-up on tasks/provects assigned to subcontractors. wal] i] a] al) A] OD 

44. Promulgation of general cumose information about what is happening around NARDAC Na] L] a_] AL) A iC] 


(i.@., new hardware, system software, application software, training, new personnel, etc.) 
through newsietters, electronic bulletin boards, and Cliemt Council meetings. 
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4 NAROAC CLIENT SATISFACTION SURVEY 


PLEASE PLACE AN X IN THE BOX WHICH BEST DESCRIBES THE RELATIVE IMPORTANCE TO YOUR 
ORGANIZATION OF EACH OF THE FOLLOWING PERFORMANCE CRITERIA AS IT RELATES TO CUENT 
SATISFACTION. 


USE THE FOLLOWING SCALE: 5 CRITICAL å IMPORTANT 3 AVERAGE 2 NOT IMPORTANT 1 IRRELEVANT 

1. Accessibility s[] aL] a] AI] N 
(Ease of obtaining service) 

2. System Reeources s «O a] L] 0 


(Encompasses hardware. software, and 
teleprocessing technology) 


3. Cost Menegemsnt d.] «O aL] Ad] UO 


(includes controls over project budgets, rates 
charged for services, billing procedures, and pnce 
Structures) 


4. Quality of Products end Servicee 
(NARDAGC's ability to deliver reliable services and 


excellent products) 


5. Timsiiness of Services 
(NARDAC's ability to be punctual and “schedule 
conscientious” wth Rs products and services) 


6. Reeponsivsness 
(How quickly NARDAC reacts or responds to 
cient needs) 


7. Stsffing 
(Encompasses the resourceiulness and requiske 
qualificavons of NARDAC personnel) 


a Res Ma 

MA AA 5 

Mo A a 

a Mo aa e 
O 


8. Gensrel Buelnsss Prsctices 
(The manner in which NARDAC conducts business 
with its clients) 





Comments: 


























Code Telephone 


j 











PLEASE PANT 
Nama Of your organcation Name of your NARDAC POC. 
Thank you for completing this survey. The enclosed envelope has been provided for your convenience. 
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APPENDIX K. REVISED OUESTIONNAIRE 


NARDAC CLIENT SATISFACTION SUAVEY 


PLEASE PLACE AN XIN THE BOX WHICH BEST DESCRIBES YOUR LEYEL OF SATISFACTION WITH EACH OF THE 
FOLLOWING ITEMS. (Additonal comments may be made on the last page). 


USE THE FOLLOWING SCALE: 


N/A NOT APPUCABLE 5 EXTREMELY SATISFED 4 SATISFED 3 NEUTRAL 2 OSSATISFIED 


1 EXTREMELY DISSATISFIED 





(Ease of obtaming service) 


ae 


The availability of your NARDAC point of contact. 


The procedures required inNAVDACINST 5290.1 for obtaining vanous NARDAC products 
and services. 


The current hours of operation at NARDAC. 


The physical location of NARDAC relative to your command. 


The quantity of vister parking spaces at NARDAC. 


System. Resources, 
(Encompasses hardware, software, and teleprocessing technology) 


6. 


The current computer hardware capabilities at NARDAC. 


(1.0.. Mainframe, minicomputers, microcomputers) 


wal] 
wal 
wal] 
wal] 
wal] 


wal] 





The ability of NARDAC to meet your ogrojected automated ¡ 


The currency and vanety of software packages available at NARDAC. 


The ability of NAROAC to provide adequate {aieorocessing services to meet your 
command's operational needs. 


(Includes controls over project budgets, rates charged for services, billing procedures, 
and price structures) 


10. 


NARDAG's ability to provide an accurate assessment of both 
the costs and the schedule for a project. 


NARDAC's adherence to oroiact budgets. 
The ability of NARDAC to adequately exolain amiect costs. 
The format ot NARDAC’s Chargeback Report (Le., easy to read and understand). 


The accuracy of NARDAC’s Chargeback Report. 
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wal] 
wal] 


wal] 


wal] 
wal] 


wal] 


AA A AÀ 
A ÀA à Ò Ò 


ai AA å 


hA AA À 


og er N O 


HO 4 4 å 


AAAA 
A À À ù Ò 


hA À 


i ee 


hA AA À 


a Aja ab 


NARDAC CLIENT SATISFACTION SURVEY 


USE THE FOLLOWING SCALE: 


NWA NOT APPLICABLE S EXTREMELY SATISFED 4 SATISFED 3 NEUTRAL 2 DISSATISFED 


1 EXTREMELY DISSATISFED 





15. The curentrates NARDAC charges for products and services. 


16. Eigxibilty in charging tor NARDAC products and services. 


(e.g.. fixed-pnce contract, tiered rate structure. etc.). 


Quality of Products and Services, 
(NARDAC's ability to deliver reliable services and excellent products) 


17. The quality ot NARDAC developed anplicaton system products. 


18. 


19. 


20. 


21. 


23. 


The quality of NARDAC's servicas. 
(Services inciude training Information Resource Center (IRC) assistance, 
processing, Client Liason servmces.studies etc.) 


The securty conptrois ever your data that is maintained on NARDAC computers. 

(i.0., physical security, user passwords, etc.) 

The reliability of NARDAC online and batch systems (i.e., the total time the system is up). 
The accuracy of NARDAC's system documentation/imanuais. 


The response time of NARDAC’s online systems. 


The quality of the resolution to a problem associated with a NARDAC product or sernce. 


Iimeliness of Services, 
(NARDAC's ability to be punctual and ° schedule conscientious” with as products and services) 


24. 


25. 


26. 


27. 


28. 


NARDAC's adherence to project schedules as specified in the 
Project Request. 


The ability of NARDAC to adhere to production pmesssina.schedules. 


The frequency of NARDAC progress repors 


The timely notification of delays in output. 


Timeliness of services from NARDAC. 
(Services inciude training, information Resource Center (IRC) assistance, 
processing,Cliemt Liaison services,studies,etc.) 


(How quickly NARDAC reacts or responds to cient needs) 


29. Software maintenance pertormed on existing NARDAC systems (ie., responsiveness in 


30. 


correcting program errors). 
The responsiveness of NARDAC to a request for services. 
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NARDAC CLIENT SATISFACTION SURVEY 


USE THE FOLLOWING SCALE: 





NA NOT APPLICABLE 5 EXTREMELY SATISFED 4 SATISFED 3 NEUTRAL 2 DISSATISFIED 1 EXTREMELY DISSATISFIED 

31. The reponsweness of NARDAC to resolving a reporied problem associated with a product na d] A] O 
Or Service. 

32. The flexibility of NARDAC to changes in your project requirements. na d «O a] A] J 

33. The level of assistance provided in preparing proposals for new projects. na -E a] a] A] iC] 

Staffing, 

(Encompasses the resourcefuiness and reguisie qualifications of NARDAC personnel) 

34. The overall range of gxparise within the NARDAC organzation. wal] I] a a] A] i) 

35. The level of expertise of the NARDAC personnel with whom you deal. wa "E a] al] a] J 

36. The ability of NARDAC personnel to keep abreast of the latest develonments in their na m 'n a] 4] 0O 
respective functional area. 

37. The couneous and professional attitude of the NARDAC personnel with whom you deal. na I] a] a] a] iL) 

38. The technica! statt's ability 10 design and develop creative and innovative systems. waL] di] a] a] a] iC] 
(Technical staff includes computer specialists, programmers, and system analysts) 

39. The technical statf's ability to satisfactorily anaivze and document your prolest val] I] a] al] A] J 
requirements. 

40. The abilty of NARDAC’s technical staff to communicate in client terns/lanquage. wal] D a] a] a] LJ 

General Business Practices, 

(The manner in which NARDAC conducts business with its clients) 

41. NARDAC's tamilarty with your command's mission as tt relates to data processing. na L a] a] A) O 

42. NARDAC's understanding of the impact on your command for late or inaccurate products na - 'n AI] 2) iL] 
Or services. 

43. The coordination and foliow-up on tasks/orojects assigned to subcontractars, wal] I] n a] L DO 

44. Promuigaton of general pumose information about what is happening around NARDAC na] I] a] a] A] 1] 


(1.0.. new hardware, system software, application software, training, new personnel, etc.) 
through newsletters, electronic bulletin boards, and Client Council meetings. 
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NARDAC CLIENT SATISFACTION SURVEY 


PLEASE PLACE AN X iN THE BOX WHICH BEST DESCRIBES THE RELATIVE IMPORTANCE TO YOUR ORGANIZATION 
OF EACH OF THE FOLLOWING PERFORMANCE CRITERIA AS IT RELATES TO CUENT SATISFACTION. 


USE THE FOLLOWING SCALE: 5 CRITICAL å IMPORTANT «3 AVERAGE 2 NOT IMPORTANT 1 IRRELEVANT 

1. Acceeeibiiity d] aO O n 0 
(Ease of obtaining service) 

2. Syetem Resources d] 4] I A) OD 


(Encompasses hardware, software, and 
teleprocessing technology) 


3. Coet Mensgement m «O 4] A] 0 


(Inciudes controls over project budgets, rates 

charged for services, billing procedures, and pnce 

structures) 

4. Quality of Products end Services DO 
(NARDAC's ability to delrver reliable services and 


excellent products) 


5. Timeliness of Services 
(NARDAC's ability to be punctual and “schedule 
conscientious® with its products and services) 


6. Reeponsiveness 
(How quickly NARDAC reacts or responds to 
cent needs) 


ed 


Steffing 
(Encompasses the resourcefulness and requiste 
gualdicauons of NARDAC personnel) 


eee O G 

a 6 o D 

a w al G 

amn D G B 
O 


Genersi Bueiness Prectices 
(The manner in which NARDAC conducts business 
with its cients) 





Comments: 


























Cade Telephone 


í 











Name of your NARDAC POC, 





Name of your organation 
Thank you for completing this survey. The enclosed envelope has been provided for your convenience. 
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